Automated testing error assessment system

ABSTRACT

Methods and systems for automatically resolving computerized electronic communication anomalies are disclosed herein. The system can include a memory including an error database containing information identifying a plurality of previous detected errors and configuration information associated with those errors. The system can include a plurality of user devices. Each of these plurality of user devices can include: a first network interface to exchange data via the communication network; and a first I/O subsystem to convert electrical signals to user interpretable outputs via a user interface. The system can include a server that can: receive an indication of the initiation of electronic communication; receive an electrical signal including attribute information; receive an error message; identify a trend in error messages; and provide an error solution if a trend is identified.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/349,937 entitled “AUTOMATED TESTING ERROR ASSESSMENT SYSTEM” andfiled on Nov. 11, 2016, which claims the benefit of U.S. ProvisionalApplication No. 62/256,099, filed on Nov. 16, 2015, and entitled“AUTOMATED TESTING ERROR ASSESSMENT SYSTEM,” the entirety of which ishereby incorporated by reference herein.

BACKGROUND OF THE INVENTION

This disclosure relates in general to machine learning and alertprovision via machine learning. Machine learning is a subfield ofcomputer science that evolved from the study of pattern recognition andcomputational learning theory in artificial intelligence. Machinelearning explores the construction and study of algorithms that canlearn from and make predictions on data. Such algorithms operate bybuilding a model from example inputs in order to make data-drivenpredictions or decisions, rather than following strictly static programinstructions.

Machine learning is closely related to and often overlaps withcomputational statistics; a discipline that also specializes inprediction-making. It has strong ties to mathematical optimization,which deliver methods, theory and application domains to the field.Machine learning is employed in a range of computing tasks wheredesigning and programming explicit, rule-based algorithms is infeasible.Example applications include spam filtering, optical characterrecognition (OCR), search engines and computer vision. Machine learningis sometimes conflated with data mining, although that focuses more onexploratory data analysis. Machine learning and pattern recognition canbe viewed as two facets of the same field. When employed in industrialcontexts, machine learning methods may be referred to as predictiveanalytics or predictive modelling.

While machine learning and alert provision via machine learning areadvantageous technologies, new methods and techniques for theapplication of machine learning and alert provisioning are desired.

BRIEF SUMMARY OF THE INVENTION

One aspect of the present disclosure relates to a system forautomatically resolving computerized electronic communication anomalies.The system includes: memory including an error database containinginformation identifying a plurality of previous detected errors andconfiguration information associated with those errors. The systemincludes a plurality of user devices. In some embodiments, some or allof the plurality of user devices includes: a first network interfacethat can exchange data via the communication network; and a first I/Osubsystem that can convert electrical signals to user interpretableoutputs via a user interface. The system can include a server. In someembodiments, the server can: receive an indication of the initiation ofelectronic communication, which indication of the initiation ofelectronic communication identifies a plurality of users of theplurality of user devices; receive an electrical signal includingattribute information identifying one or several attributes of each ofthe user devices; receive an error message indicating a problem in theelectronic communication; identify a trend in error messages bycomparing the received error message to gathered error data; and providean error solution if a trend is identified.

In some embodiments, the memory further includes a user profile databasecontaining information identifying one or several attributes of a user.In some embodiments, the attribute information includes locationinformation, which location information identifies the location of eachof the plurality of user devices. In some embodiments, the electroniccommunication includes test communications. In some embodiments, thetest communications include a plurality of questions and a plurality ofresponses to the plurality of questions.

In some embodiments, the attribute information includes hardwareinformation, which hardware information identifies hardware of each ofthe plurality of user devices. In some embodiments, the attributeinformation includes software information, which software informationidentifies software on each of the plurality of user devices. In someembodiments, the attribute information includes an event log identifyingoperations performed by each of the plurality of user devices before theproblem in electronic communication and identifying software running oneach of the plurality of user devices at the time of the problem in theelectronic communication.

In some embodiments, providing an error solution includes generating andsending an alert, which alert is sent from the server to the userdevice. In some embodiments, the alert launches an application withinthe user device, which application displays data contained in the alert.In some embodiments, the alert includes code to direct the launch of theapplication at the user device.

One aspect of the present disclosure relates to a method ofautomatically resolving computerized electronic communication anomalies.The method includes: receiving at a server an indication of theinitiation of electronic test from a user device, which indication ofthe initiation of testing identifies a plurality of testers; receivingat the server an electrical signal including attribute informationidentifying one or several attributes of each of the user device fromthe user device; receiving at the server an error message indicating aproblem in the testing from the user device; retrieving gathered errordata from a memory including an error database including informationidentifying a plurality of previous detected errors and configurationinformation associated with those errors; comparing the received errormessage to the gathered error data; identifying a trend in the errormessage and the gathered error data based on the comparison of thereceived error message to the gathered error data; and providing anerror solution if a trend is identified.

In some embodiments, the attribute information includes locationinformation, which location information identifies the location of eachof the plurality of user devices. In some embodiments, the electroniccommunication includes test communications. In some embodiments, thetest communications include a plurality of questions and a plurality ofresponses to the plurality of questions.

In some embodiments, the attribute information includes hardwareinformation, which hardware information identifies hardware of each ofthe plurality of user devices. In some embodiments, the attributeinformation includes software information, which software informationidentifies software on each of the plurality of user devices. In someembodiments, the attribute information includes an event log identifyingoperations performed by each of the plurality of user devices before theproblem in electronic communication and identifying software running oneach of the plurality of user devices at the time of the problem in theelectronic communication.

In some embodiments, providing an error solution includes generating andsending an alert. In some embodiments, the alert is sent from the serverto the user device. In some embodiments, the alert launches anapplication within the user device, which application displays datacontained in the alert.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is described in conjunction with the appendedfigures:

FIG. 1 is a block diagram showing illustrating an example of a contentdistribution network.

FIG. 2 is a block diagram illustrating a computer server and computingenvironment within a content distribution network.

FIG. 3 is a block diagram illustrating an embodiment of one or more datastore servers within a content distribution network.

FIG. 4A is a block diagram illustrating an embodiment of one or morecontent management servers within a content distribution network.

FIG. 4B is a flowchart illustrating one embodiment of a process for datamanagement.

FIG. 4C is a flowchart illustrating one embodiment of a process forevaluating a response.

FIG. 5 is a block diagram illustrating the physical and logicalcomponents of a special-purpose computer device within a contentdistribution network.

FIG. 6 is a block diagram illustrating one embodiment of thecommunication network.

FIG. 7 is a block diagram illustrating one embodiment of user device andsupervisor device communication.

FIG. 8 is a block diagram illustrating one embodiment of networkeddevices including a user device and a supervisor device.

FIG. 9 is a schematic illustration of one embodiment of a user devicefor use with a content distribution network.

FIG. 10 is a first part of a flowchart illustrating one embodiment of aprocess for automatic electronic communication error detection andremediation.

FIG. 11 is a second part of the flowchart illustrating one embodiment ofthe process for automatic electronic communication error detection andremediation

FIG. 12 is a flowchart illustrating one embodiment of a process forgenerating a performance adjustment based on a detected electroniccommunication error.

FIG. 13 is a flowchart illustrating one embodiment of a process forreceiving electronic communication system performance feedback.

In the appended figures, similar components and/or features may have thesame reference label. Where the reference label is used in thespecification, the description is applicable to any one of the similarcomponents having the same reference label. Further, various componentsof the same type may be distinguished by following the reference labelby a dash and a second label that distinguishes among the similarcomponents. If only the first reference label is used in thespecification, the description is applicable to any one of the similarcomponents having the same first reference label irrespective of thesecond reference label.

DETAILED DESCRIPTION OF THE INVENTION

The ensuing description provides illustrative embodiment(s) only and isnot intended to limit the scope, applicability or configuration of thedisclosure. Rather, the ensuing description of the illustrativeembodiment(s) will provide those skilled in the art with an enablingdescription for implementing a preferred exemplary embodiment. It isunderstood that various changes can be made in the function andarrangement of elements without departing from the spirit and scope asset forth in the appended claims.

With reference now to FIG. 1, a block diagram is shown illustratingvarious components of a content distribution network (CDN) 100 whichimplements and supports certain embodiments and features describedherein. Content distribution network 100 may include one or more contentmanagement servers 102. As discussed below in more detail, contentmanagement servers 102 may be any desired type of server including, forexample, a rack server, a tower server, a miniature server, a bladeserver, a mini rack server, a mobile server, an ultra-dense server, asuper server, or the like, and may include various hardware components,for example, a motherboard, a processing units, memory systems, harddrives, network interfaces, power supplies, etc. Content managementserver 102 may include one or more server farms, clusters, or any otherappropriate arrangement and/or combination or computer servers. Contentmanagement server 102 may act according to stored instructions locatedin a memory subsystem of the server 102, and may run an operatingsystem, including any commercially available server operating systemand/or any other operating systems discussed herein.

The content distribution network 100 may include one or more data storeservers 104, also referred to herein as “databases”, such as databaseservers and/or file-based storage systems. The database servers 104 canaccess data that can be stored on a variety of hardware components.These hardware components can include, for example, components formingtier 0 storage, components forming tier 1 storage, components formingtier 2 storage, and/or any other tier of storage. In some embodiments,tier 0 storage refers to storage that is the fastest tier of storage inthe database server 104, and particularly, the tier 0 storage is thefastest storage that is not RAM or cache memory. In some embodiments,the tier 0 memory can be embodied in solid state memory such as, forexample, a solid-state drive (SSD) and/or flash memory.

In some embodiments, the tier 1 storage refers to storage that is one orseveral higher performing systems in the memory management system, andthat is relatively slower than tier 0 memory, and relatively faster thanother tiers of memory. The tier 1 memory can be one or several harddisks that can be, for example, high-performance hard disks. These harddisks can be one or both of physically or communicatingly connected suchas, for example, by one or several fiber channels. In some embodiments,the one or several disks can be arranged into a disk storage system, andspecifically can be arranged into an enterprise class disk storagesystem. The disk storage system can include any desired level ofredundancy to protect data stored therein, and in one embodiment, thedisk storage system can be made with grid architecture that createsparallelism for uniform allocation of system resources and balanced datadistribution.

In some embodiments, the tier 2 storage refers to storage that includesone or several relatively lower performing systems in the memorymanagement system, as compared to the tier 1 and tier 2 storages. Thus,tier 2 memory is relatively slower than tier 1 and tier 0 memories. Tier2 memory can include one or several SATA-drives or one or severalNL-SATA drives.

In some embodiments, the one or several hardware and/or softwarecomponents of the database server 104 can be arranged into one orseveral storage area networks (SAN), which one or several storage areanetworks can be one or several dedicated networks that provide access todata storage, and particularly that provides access to consolidated,block level data storage. A SAN typically has its own network of storagedevices that are generally not accessible through the local area network(LAN) by other devices. The SAN allows access to these devices in amanner such that these devices appear to be locally attached to the userdevice.

Databases 104 may comprise stored data relevant to the functions of thecontent distribution network 100. Illustrative examples of databases 104that may be maintained in certain embodiments of the contentdistribution network 100 are described below in reference to FIG. 3. Insome embodiments, multiple databases may reside on a single databaseserver 104, either using the same storage components of server 104 orusing different physical storage components to assure data security andintegrity between databases. In other embodiments, each database mayhave a separate dedicated database server 104.

The content distribution network 100 can include one or several end-usernetworks 107. In the embodiment shown in FIG. 1, the contentdistribution network 100 includes a first end-user network 107-A and asecond end-user network 107-B. In some embodiments, one or both of thefirst and second end-user networks 107-A, 107-B can include the same ordifferent components. In the following discussion, and to the extentthat a component is generally discussed, the components of the one orseveral end-user networks 107 are identified without specifying whetherthey belong to the first or second end-user network 107-A, 107-B.

In some embodiments, the first and second end-user networks 107-A, 107-Bcan be at different locations, such as different geographic locations,can be controlled by the same entity or by different entities, canrepresent political divisions, or the like. In one embodiment, forexample, the first end-user network 107-A can be controlled by a firstschool or school district, and the second end-user network 107-B can becontrolled by a second school or school district.

The end-user network 107 can be connected with other components of thecontent distribution network 100 and/or with other networks within thecontent distribution network 100. The end-user network 107 of thecontent distribution network 100 also may include one or more userdevices 106 and/or supervisor devices 110. User devices 106 andsupervisor devices 110 may display content received via the contentdistribution network 100, and may support various types of userinteractions with the content. In some embodiments, the user devices 106and the supervisor devices 110 can be configured for computerizedtesting, and can provide testing content to users of those devices 106,110. In some embodiments, the computerized testing can include, forexample, high-stakes testing, standardized testing, quizzing, or thelike. Additionally, in some embodiments, these devices 106, 110 can beconfigured to track their performance in providing the computerized testto students and can provide data relating to this performance to othercomponents of the content distribution network 100.

User devices 106 and supervisor devices 110 may include mobile devicessuch as smartphones, tablet computers, personal digital assistants, andwearable computing devices. Such mobile devices may run a variety ofmobile operating systems, and may be enabled for Internet, e-mail, shortmessage service (SMS), Bluetooth®, mobile radio-frequency identification(M-RFID), and/or other communication protocols. Other user devices 106and supervisor devices 110 may be general purpose personal computers orspecial-purpose computing devices including, by way of example, personalcomputers, laptop computers, workstation computers, projection devices,and interactive room display systems. Additionally, user devices 106 andsupervisor devices 110 may be any other electronic devices, such as athin-client computers, an Internet-enabled gaming system, business orhome appliances, and/or a personal messaging devices capable ofcommunicating over network(s) 120.

In different contexts of content distribution networks 100, user devices106 and supervisor devices 110 may correspond to different types ofspecialized devices, for example, student devices and teacher devices inan educational network, employee devices and presentation devices in acompany network, different gaming devices in a gaming network, etc. Insome embodiments, user devices 106 and supervisor devices 110 mayoperate in the same physical location, such as a classroom or conferenceroom. In such cases, the devices may contain components that supportdirect communications with other nearby devices, such as a wirelesstransceivers and wireless communications interfaces, Ethernet sockets orother Local Area Network (LAN) interfaces, etc. In otherimplementations, the user devices 106 and supervisor devices 110 neednot be used at the same location, but may be used in remote geographiclocations in which each user device 106 and supervisor device 110 mayuse security features and/or specialized hardware (e.g.,hardware-accelerated SSL and HTTPS, WS-Security, firewalls, etc.) tocommunicate with the content management server 102 and/or other remotelylocated user devices 106. Additionally, different user devices 106 andsupervisor devices 110 may be assigned different designated roles, suchas presenter devices, teacher devices, administrator devices, or thelike, and in such cases the different devices may be provided withadditional hardware and/or software components to provide content andsupport user capabilities not available to the other devices.

The end-user network 107 of the content distribution network 100 canfurther include one or several end-user servers 105 and/or one orseveral local data servers 109 and/or local memories. In someembodiments, the one or several end-user servers 107 can be any desiredtype of server including, for example, a rack server, a tower server, aminiature server, a blade server, a mini rack server, a mobile server,an ultra-dense server, a super server, or the like, and may includevarious hardware components, for example, a motherboard, a processingunits, memory systems, hard drives, network interfaces, power supplies,etc. End-user server 105 may include one or more server farms, clusters,or any other appropriate arrangement and/or combination or computerservers. End-user server 105 may act according to stored instructionslocated in a memory subsystem of the End-user server 105, and may run anoperating system, including any commercially available server operatingsystem and/or any other operating systems discussed herein.

The end-user server 105 can be configured to receive content from acontent network 122, discussed below, and provide that content to theone or several user devices 106. In some embodiments, the end-userserver 105 can be further configured to control and/or direct theoperation of some or all of the user devices 106 and supervisor device110. This can include, for example, directing some or all of the userdevices 106 to begin testing and/or to begin providing testing contentto their respective users, to end testing and/or to stop providingtesting content to their respective users, or the like.

The local data server 109 can be a database server similar to databaseserver 104, and can include, for example, some or all of the hardwareand/or software components of the database server 104. The local dataserver 109 can be configured to store and/or can store a local copy ofthe content received from the content network 122. The local data server109 can be initially configured to store one or several communicationsreceived from the content network 122 and/or provided to the contentnetwork 122. In some embodiments, the local data server 109 can providea local cache for recent data.

The content distribution network 100 also may include a privacy server108 that maintains private user information at the privacy server 108while using applications or services hosted on other servers. Forexample, the privacy server 108 may be used to maintain private data ofa user within one jurisdiction even though the user is accessing anapplication hosted on a server (e.g., the content management server 102)located outside the jurisdiction. In such cases, the privacy server 108may intercept communications between a user device 106 or supervisordevice 110 and other devices that include private user information. Theprivacy server 108 may create a token or identifier that does notdisclose the private information and may use the token or identifierwhen communicating with the other servers and systems, instead of usingthe user's private information.

As illustrated in FIG. 1, the content management server 102 may be incommunication with one or more additional servers, such as a contentserver 112, a user data server 112, and/or an administrator server 116.Each of these servers may include some or all of the same physical andlogical components as the content management server(s) 102, and in somecases, the hardware and software components of these servers 112-116 maybe incorporated into the content management server(s) 102, rather thanbeing implemented as separate computer servers.

Content server 112 may include hardware and software components togenerate, store, and maintain the content resources for distribution touser devices 106 and other devices in the network 100. For example, incontent distribution networks 100 used for professional training andeducational purposes, content server 112 may include databases oftraining materials, presentations, plans, syllabi, reviews, evaluations,interactive programs and simulations, course models, course outlines,and various training interfaces that correspond to different materialsand/or different types of user devices 106. In content distributionnetworks 100 used for media distribution, interactive gaming, and thelike, a content server 112 may include media content files such asmusic, movies, television programming, games, and advertisements.

User data server 114 may include hardware and software components thatstore and process data for multiple users relating to each user'sactivities and usage of the content distribution network 100. Forexample, the content management server 102 may record and track eachuser's system usage, including their user device 106, content resourcesaccessed, and interactions with other user devices 106. This data may bestored and processed by the user data server 114, to support usertracking and analysis features. For instance, in the professionaltraining and educational contexts, the user data server 114 may storeand analyze each user's training materials viewed, presentationsattended, courses completed, interactions, evaluation results, and thelike. The user data server 114 may also include a repository foruser-generated material, such as evaluations and tests completed byusers, and documents and assignments prepared by users. In the contextof media distribution and interactive gaming, the user data server 114may store and process resource access data for multiple users (e.g.,content titles accessed, access times, data usage amounts, gaminghistories, user devices and device types, etc.).

Administrator server 116 may include hardware and software components toinitiate various administrative functions at the content managementserver 102 and other components within the content distribution network100. For example, the administrator server 116 may monitor device statusand performance for the various servers, databases, and/or user devices106 in the content distribution network 100. When necessary, theadministrator server 116 may add or remove devices from the network 100,and perform device maintenance such as providing software updates to thedevices in the network 100. Various administrative tools on theadministrator server 116 may allow authorized users to set user accesspermissions to various content resources, monitor resource usage byusers and devices 106, and perform analyses and generate reports onspecific network users and/or devices (e.g., resource usage trackingreports, training evaluations, etc.).

The content distribution network 100 may include one or several errorservers 119. The one or several error servers 119 may be any desiredtype of server including, for example, a rack server, a tower server, aminiature server, a blade server, a mini rack server, a mobile server,an ultra-dense server, a super server, or the like, and may includevarious hardware components, for example, a motherboard, a processingunits, memory systems, hard drives, network interfaces, power supplies,etc. Error servers 119 may include one or more server farms, clusters,or any other appropriate arrangement and/or combination or computerservers. Error servers 119 may act according to stored instructionslocated in a memory subsystem of the error servers 119, and may run anoperating system, including any commercially available server operatingsystem and/or any other operating systems discussed herein.

The one or several error servers 119 can be configured to receive one orseveral communications from one or both of the first and second end-usernetworks 107-A, 107-B and determine the existence of a testing anomaly.In some embodiments, this testing anomaly can be an anomaly thatadversely affects the ability of one or several of the user devices 106to provide testing content to their respective users. These anomaliescan arise from a variety of different sources including, for example,network problems, hardware, software, hardware or softwareconfiguration, or the like. In some embodiments, the one or severalerror servers 119 can be configured to gather data relating to one orseveral testing anomalies and identifies correspondence between theseanomalies. Some examples of a correspondence between anomalies includelocation of user devices experiencing anomalies in the same geographicregion, in the same end-user network 107, having the same hardware orsoftware, the same hardware or software configuration, the same network,the providing of the same testing content, the same type of error, thesame error effect, or the like.

Once the correspondence between the anomalies has been detected, the oneor several error servers 119 can be configured to generate a remediationrecommendation and/or identify a solution to the source of the anomaly.In some embodiments, this can also include generating and/or identifyingan adjustment value with which testing scores of students affected bythe anomaly can be adjusted.

The content distribution network 100 may include one or morecommunication networks 120. Although only a single network 120 isidentified in FIG. 1, the content distribution network 100 may includeany number of different communication networks between any of thecomputer servers and devices shown in FIG. 1 and/or other devicesdescribed herein. Communication networks 120 may enable communicationbetween the various computing devices, servers, and other components ofthe content distribution network 100. As discussed below, variousimplementations of content distribution networks 100 may employdifferent types of networks 120, for example, computer networks,telecommunications networks, wireless networks, and/or any combinationof these and/or other networks.

In some embodiments, some of the components of the content distributionnetwork 100 can belong to the content network 122. The content network122 can include, for example, the content management server 102, thedatabase server 104, the privacy server 108, the content server 112, theuser data server 114, the administrator server 116, and/or thecommunication network 120. The content network 122 can be the source ofcontent distributed by the content distribution network 100, whichcontent can include, for example, one or several documents and/orapplications or programs. These documents and/or applications orprograms are digital content. In some embodiments, these one or severaldocuments and/or applications or programs can include, for example, oneor several webpages, presentations, papers, videos, charts, graphs,books, written work, figures, images, graphics, recordings, applets,scripts, or the like.

The content distribution network 100 may include one or severalnavigation systems or features including, for example, the GlobalPositioning System (“GPS”), GALILEO, or the like, or location systems orfeatures including, for example, one or several transceivers that candetermine location of the one or several components of the contentdistribution network 100 via, for example, triangulation. All of theseare depicted as navigation system 124.

In some embodiments, navigation system 124 can include or severalfeatures that can communicate with one or several components of thecontent distribution network 100 including, for example, with one orseveral of the user devices 106 and/or with one or several of thesupervisor devices 110. In some embodiments, this communication caninclude the transmission of a signal from the navigation system 124which signal is received by one or several components of the contentdistribution network 100 and can be used to determine the location ofthe one or several components of the content distribution network 100.

With reference to FIG. 2, an illustrative distributed computingenvironment 200 is shown including a computer server 202, four clientcomputing devices 206, and other components that may implement certainembodiments and features described herein. In some embodiments, theserver 202 may correspond to the content management server 102 discussedabove in FIG. 1, and the client computing devices 206 may correspond tothe user devices 106. However, the computing environment 200 illustratedin FIG. 2 may correspond to any other combination of devices and serversconfigured to implement a client-server model or other distributedcomputing architecture.

Client devices 206 may be configured to receive and execute clientapplications over one or more networks 220. Such client applications maybe web browser based applications and/or standalone softwareapplications, such as mobile device applications. Server 202 may becommunicatively coupled with the client devices 206 via one or morecommunication networks 220. Client devices 206 may receive clientapplications from server 202 or from other application providers (e.g.,public or private application stores). Server 202 may be configured torun one or more server software applications or services, for example,web-based or cloud-based services, to support content distribution andinteraction with client devices 206. Users operating client devices 206may in turn utilize one or more client applications (e.g., virtualclient applications) to interact with server 202 to utilize the servicesprovided by these components.

Various different subsystems and/or components 204 may be implemented onserver 202. Users operating the client devices 206 may initiate one ormore client applications to use services provided by these subsystemsand components. The subsystems and components within the server 202 andclient devices 206 may be implemented in hardware, firmware, software,or combinations thereof. Various different system configurations arepossible in different distributed computing systems 200 and contentdistribution networks 100. The embodiment shown in FIG. 2 is thus oneexample of a distributed computing system and is not intended to belimiting.

Although exemplary computing environment 200 is shown with four clientcomputing devices 206, any number of client computing devices may besupported. Other devices, such as specialized sensor devices, etc., mayinteract with client devices 206 and/or server 202.

As shown in FIG. 2, various security and integration components 208 maybe used to send and manage communications between the server 202 anduser devices 206 over one or more communication networks 220. Thesecurity and integration components 208 may include separate servers,such as web servers and/or authentication servers, and/or specializednetworking components, such as firewalls, routers, gateways, loadbalancers, and the like. In some cases, the security and integrationcomponents 208 may correspond to a set of dedicated hardware and/orsoftware operating at the same physical location and under the controlof same entities as server 202. For example, components 208 may includeone or more dedicated web servers and network hardware in a datacenteror a cloud infrastructure. In other examples, the security andintegration components 208 may correspond to separate hardware andsoftware components which may be operated at a separate physicallocation and/or by a separate entity.

Security and integration components 208 may implement various securityfeatures for data transmission and storage, such as authenticating usersand restricting access to unknown or unauthorized users. In variousimplementations, security and integration components 208 may provide,for example, a file-based integration scheme or a service-basedintegration scheme for transmitting data between the various devices inthe content distribution network 100. Security and integrationcomponents 208 also may use secure data transmission protocols and/orencryption for data transfers, for example, File Transfer Protocol(FTP), Secure File Transfer Protocol (SFTP), and/or Pretty Good Privacy(PGP) encryption.

In some embodiments, one or more web services may be implemented withinthe security and integration components 208 and/or elsewhere within thecontent distribution network 100. Such web services, includingcross-domain and/or cross-platform web services, may be developed forenterprise use in accordance with various web service standards, such asRESTful web services (i.e., services based on the Representation StateTransfer (REST) architectural style and constraints), and/or webservices designed in accordance with the Web Service Interoperability(WS-I) guidelines. Some web services may use the Secure Sockets Layer(SSL) or Transport Layer Security (TLS) protocol to provide secureconnections between the server 202 and user devices 206. SSL or TLS mayuse HTTP or HTTPS to provide authentication and confidentiality. Inother examples, web services may be implemented using REST over HTTPSwith the OAuth open standard for authentication, or using theWS-Security standard, which provides for secure SOAP messages using XMLencryption. In other examples, the security and integration components208 may include specialized hardware for providing secure web services.For example, security and integration components 208 may include securenetwork appliances having built-in features such as hardware-acceleratedSSL and HTTPS, WS-Security, and firewalls. Such specialized hardware maybe installed and configured in front of any web servers, so that anyexternal devices may communicate directly with the specialized hardware.

Communication network(s) 220 may be any type of network familiar tothose skilled in the art that can support data communications using anyof a variety of commercially-available protocols, including withoutlimitation, TCP/IP (transmission control protocol/Internet protocol),SNA (systems network architecture), IPX (Internet packet exchange),Secure Sockets Layer (SSL) or Transport Layer Security (TLS) protocols,Hyper Text Transfer Protocol (HTTP) and Secure Hyper Text TransferProtocol (HTTPS), Bluetooth®, Near Field Communication (NFC), and thelike. Merely by way of example, network(s) 220 may be local areanetworks (LAN), such as one based on Ethernet, Token-Ring and/or thelike. Network(s) 220 also may be wide-area networks, such as theInternet. Networks 220 may include telecommunication networks such as apublic switched telephone networks (PSTNs), or virtual networks such asan intranet or an extranet. Infrared and wireless networks (e.g., usingthe Institute of Electrical and Electronics (IEEE) 802.11 protocol suiteor other wireless protocols) also may be included in networks 220.

Computing environment 200 also may include one or more databases 210and/or back-end servers 212. In certain examples, the databases 210 maycorrespond to database server(s) 104, the local data server 109, and/orthe customizer data server 128 discussed above in FIG. 1, and back-endservers 212 may correspond to the various back-end servers 112-116.Databases 210 and servers 212 may reside in the same datacenter or mayoperate at a remote location from server 202. In some cases, one or moredatabases 210 may reside on a non-transitory storage medium within theserver 202. Other databases 210 and back-end servers 212 may be remotefrom server 202 and configured to communicate with server 202 via one ormore networks 220. In certain embodiments, databases 210 and back-endservers 212 may reside in a storage-area network (SAN), or may usestorage-as-a-service (STaaS) architectural model. In some embodiments,the computing environment can be replicated for each of the networks107, 122, 104 discussed with respect to FIG. 1 above.

With reference to FIG. 3, an illustrative set of databases and/ordatabase servers is shown, corresponding to the databases servers 104 ofthe content distribution network 100 discussed above in FIG. 1. One ormore individual databases 301-315 may reside in storage on a singlecomputer server 104 (or a single server farm or cluster) under thecontrol of a single entity, or may reside on separate servers operatedby different entities and/or at remote locations. In some embodiments,databases 301-315 may be accessed by the content management server 102and/or other devices and servers within the network 100 (e.g., userdevices 106, supervisor devices 110, administrator servers 116, etc.).Access to one or more of the databases 301-315 may be limited or deniedbased on the processes, user credentials, and/or devices attempting tointeract with the database.

The paragraphs below describe examples of specific databases that may beimplemented within some embodiments of a content distribution network100. It should be understood that the below descriptions of databases301-315, including their functionality and types of data stored therein,are illustrative and non-limiting. Database server architecture, design,and the execution of specific databases 301-315 may depend on thecontext, size, and functional requirements of a content distributionnetwork 100. For example, in content distribution systems 100 used forprofessional training and educational purposes, separate databases orfile-based storage systems may be implemented in database server(s) 104to store trainee and/or student data, trainer and/or professor data,training module data and content descriptions, training results,evaluation data, and the like. In contrast, in content distributionsystems 100 used for media distribution from content providers tosubscribers, separate databases may be implemented in database server(s)104 to store listing of available content titles and descriptions,content title usage statistics, subscriber profiles, account data,payment data, network usage statistics, etc.

A user profile data store 301, also referred to herein as a user profiledatabase 301 may include information relating to the end users withinthe content distribution network 100. This information may include usercharacteristics such as the user names, access credentials (e.g., loginsand passwords), user preferences, and information relating to anyprevious user interactions within the content distribution network 100(e.g., requested content, posted content, content modules completed,training scores or evaluations, other associated users, etc.). In someembodiments, this information can relate to one or several individualend users such as, for example, one or several students, contentauthors, teachers, administrators, or the like, and in some embodiments,this information can relate to one or several institutional end userssuch as, for example, one or several schools, groups of schools such asone or several school districts, one or several colleges, one or severaluniversities, one or several training providers, or the like.

In some embodiments, this information can identify one or several usermemberships in one or several groups such as, for example, a student'smembership in a university, school, program, grade, course, class, orthe like.

In some embodiments, the user profile database 301 can includeinformation relating to a user's status, location, or the like. Thisinformation can identify, for example, a device a user is using, thelocation of that device, or the like. In some embodiments, thisinformation can be generated based on any location detection technologyincluding, for example, a navigation system 124, or the like.

Information relating to the user's status can identify, for example,logged-in status information that can indicate whether the user ispresently logged-in to the content distribution network 100 and/orwhether the log-in-is active. In some embodiments, the informationrelating to the user's status can identify whether the user is currentlyaccessing content and/or participating in an activity from the contentdistribution network 100.

In some embodiments, information relating to the user's status canidentify, for example, one or several attributes of the user'sinteraction with the content distribution network 100, and/or contentdistributed by the content distribution network 100. This can includedata identifying the user's interactions with the content distributionnetwork 100, the content consumed by the user through the contentdistribution network 100, or the like. In some embodiments, this caninclude data identifying the type of information accessed through thecontent distribution network 100 and/or the type of activity performedby the user via the content distribution network 100, the lapsed timesince the last time the user accessed content and/or participated in anactivity from the content distribution network 100, or the like. In someembodiments, this information can relate to a content program comprisingan aggregate of data, content, and/or activities, and can identify, forexample, progress through the content program, or through the aggregateof data, content, and/or activities forming the content program. In someembodiments, this information can track, for example, the amount of timesince participation in and/or completion of one or several types ofactivities, the amount of time since communication with one or severalsupervisors and/or supervisor devices 110, or the like.

In some embodiments in which the one or several end users areindividuals, and specifically are students, the user profile database301 can further include information relating to these students' academicand/or educational history. This information can identify one or severalcourses of study that the student has initiated, completed, and/orpartially completed, as well as grades received in those courses ofstudy. In some embodiments, the student's academic and/or educationalhistory can further include information identifying student performanceon one or several tests, quizzes, and/or assignments. In someembodiments, this information can be stored in a tier of memory that isnot the fastest memory in the content delivery network 100.

The user profile database 301 can include information relating to one orseveral student learning preferences. In some embodiments, for example,the user, also referred to herein as the student or the student-user mayhave one or several preferred learning styles, one or several mosteffective learning styles, and/or the like. In some embodiments, thestudent's learning style can be any learning style describing how thestudent best learns or how the student prefers to learn. In oneembodiment, these learning styles can include, for example,identification of the student as an auditory learner, as a visuallearner, and/or as a tactile learner. In some embodiments, the dataidentifying one or several student learning styles can include dataidentifying a learning style based on the student's educational historysuch as, for example, identifying a student as an auditory learner whenthe student has received significantly higher grades and/or scores onassignments and/or in courses favorable to auditory learners. In someembodiments, this information can be stored in a tier of memory that isnot the fastest memory in the content distribution network 100.

In some embodiments, the user profile database 301 can includeinformation relating to one or several student-user behavioursincluding, for example: attendance in one or several courses; attendanceand/or participation in one or several study groups; extramural, studentgroup, and/or club involve and/or participation, or the like. In someembodiments, this information relating to one or several student-userbehaviours can include information relating to the student-usersschedule.

The user profile database 301 can further include information relatingto one or several teachers and/or instructors who are responsible fororganizing, presenting, and/or managing the presentation of informationto the student. In some embodiments, user profile database 301 caninclude information identifying courses and/or subjects that have beentaught by the teacher, data identifying courses and/or subjectscurrently taught by the teacher, and/or data identifying courses and/orsubjects that will be taught by the teacher. In some embodiments, thiscan include information relating to one or several teaching styles ofone or several teachers. In some embodiments, the user profile database301 can further include information indicating past evaluations and/orevaluation reports received by the teacher. In some embodiments, theuser profile database 301 can further include information relating toimprovement suggestions received by the teacher, training received bythe teacher, continuing education received by the teacher, and/or thelike. In some embodiments, this information can be stored in a tier ofmemory that is not the fastest memory in the content distributionnetwork 100.

An accounts datastore 302, also referred to herein as an accountsdatabase 302, may generate and store account data for different users invarious roles within the content distribution network 100. For example,accounts may be created in an accounts database 302 for individual endusers, supervisors, administrator users, and entities such as companiesor educational institutions. Account data may include account types,current account status, account characteristics, and any parameters,limits, restrictions associated with the accounts.

A content library datastore 303, also referred to herein as a contentlibrary database 303, may include information describing the individualcontent items (or content resources or data packets) available via thecontent distribution network 100. In some embodiments, the librarydatabase 303 may include metadata, properties, and other characteristicsassociated with the content resources stored in the content server 112.In some embodiments, this data can include the one or several items thatcan include one or several documents and/or one or several applicationsor programs. In some embodiments, the one or several items can include,for example, one or several webpages, presentations, papers, videos,charts, graphs, books, written work, figures, images, graphics,recordings, or any other document, or any desired software orapplication or component thereof including, for example, a graphicaluser interface (GUI), all or portions of a Learning Management System(LMS), all or portions of a Content Management System (CMS), all orportions of a Student Information Systems (SIS), or the like.

In some embodiments, the data in the content library database 303 mayidentify one or more aspects or content attributes of the associatedcontent resources, for example, subject matter, access level, or skilllevel of the content resources, license attributes of the contentresources (e.g., any limitations and/or restrictions on the licensableuse and/or distribution of the content resource), price attributes ofthe content resources (e.g., a price and/or price structure fordetermining a payment amount for use or distribution of the contentresource), rating attributes for the content resources (e.g., dataindicating the evaluation or effectiveness of the content resource), andthe like. In some embodiments, the library database 303 may beconfigured to allow updating of content metadata or properties, and toallow the addition and/or removal of information relating to the contentresources. In some embodiments, the content library database 303 can beorganized such that content is associated with one or several coursesand/or programs in which the content is used and/or provided. In someembodiments, the content library database 303 can further include one orseveral teaching materials used in the course, a syllabus, one orseveral practice problems, one or several tests, one or several quizzes,one or several assignments, or the like. All or portions of the contentlibrary database can be stored in a tier of memory that is not thefastest memory in the content distribution network 100. For example,content relationships may be implemented as graph structures, which maybe stored in the library data store 303 or in an additional store foruse by selection algorithms along with the other metadata.

In some embodiments, the content library database 303 can compriseinformation to facilitate in authoring new content. This information cancomprise, for example, one or several specifications identifyingattributes and/or requirements of desired newly authored content. Insome embodiments, for example, a content specification can identify oneor several of a subject matter; length, difficulty level, or the likefor desired newly authored content.

In some embodiments, the content library database 303 can furtherinclude information for use in evaluating newly authored content. Insome embodiments, this evaluation can comprise a determination ofwhether and/or the degree to which the newly authored contentcorresponds to the content specification, or some or all of therequirements of the content specification. In some embodiments, thisinformation for use in evaluation newly authored content can identify ordefine one or several difficulty levels and/or can identify or defineone or several acceptable difficulty levels. In some embodiments, forexample, this information for use in evaluation newly authored contentcan define a plurality of difficulty levels and can delineate betweenthese difficulty levels, and in some embodiments, this information foruse in evaluation newly authored content can identify which of thedefined difficulty levels are acceptable. In other embodiments, thisinformation for use in evaluation newly authored content can merelyinclude one or several definitions of acceptable difficulty levels,which acceptable difficulty level can be based on one or severalpre-existing difficult measures such as, for example, an Item ResponseTheory (IRT) value such as, for example, an IRT b value, a p valueindicative of the proportion of correct responses in a set of responses,a grade level, or the like.

In some embodiments, this information for use in evaluation newlyauthored content can further define one or several differentiationand/or discrimination levels and/or define one or several acceptabledifferentiation and/or discrimination levels or ranges. As used herein,“differentiation” and “discrimination” refer to the degree to which anitem such as a question identifies low ability versus high abilityusers. In some embodiments, this information for use in evaluation newlyauthored content can identify one or several acceptable levels and/orranges of discrimination which levels and/or ranges can be based on oneor several currently existing discrimination measures such as, forexample, a Point-Biserial Correlation.

A pricing database 304 may include pricing information and/or pricingstructures for determining payment amounts for providing access to thecontent distribution network 100 and/or the individual content resourceswithin the network 100. In some cases, pricing may be determined basedon a user's access to the content distribution network 100, for example,a time-based subscription fee, or pricing based on network usage and. Inother cases, pricing may be tied to specific content resources. Certaincontent resources may have associated pricing information, whereas otherpricing determinations may be based on the resources accessed, theprofiles and/or accounts of the users, and the desired level of access(e.g., duration of access, network speed, etc.). Additionally, thepricing database 304 may include information relating to compilationpricing for groups of content resources, such as group prices and/orprice structures for groupings of resources.

A license database 305 may include information relating to licensesand/or licensing of the content resources within the contentdistribution network 100. For example, the license database 305 mayidentify licenses and licensing terms for individual content resourcesand/or compilations of content resources in the content server 112, therights holders for the content resources, and/or common or large-scaleright holder information such as contact information for rights holdersof content not included in the content server 112.

A content access database 306 may include access rights and securityinformation for the content distribution network 100 and specificcontent resources. For example, the content access database 306 mayinclude login information (e.g., user identifiers, logins, passwords,etc.) that can be verified during user login attempts to the network100. The content access database 306 also may be used to store assignedroles and/or levels of access to users. For example, a user's accesslevel may correspond to the sets of content resources and/or the clientor server applications that the user is permitted to access. Certainusers may be permitted or denied access to certain applications andresources based on their subscription level, training program,course/grade level, etc. Certain users may have supervisory access overone or more end users, allowing the supervisor to access all or portionsof the end user's content, activities, evaluations, etc. Additionally,certain users may have administrative access over some users and/or someapplications in the content management network 100, allowing such usersto add and remove user accounts, modify user access permissions, performmaintenance updates on software and servers, etc.

A source data store 307 may include information relating to the sourceof the content resources available via the content distribution network.For example, a source data store 307 may identify the authors andoriginating devices of content resources, previous pieces of data and/orgroups of data originating from the same authors or originating devices,and the like.

An evaluation data store 308 may include information used to direct theevaluation of users and content resources in the content managementnetwork 100. In some embodiments, the evaluation data store 308 maycontain, for example, the analysis criteria and the analysis guidelinesfor evaluating users (e.g., trainees/students, gaming users, mediacontent consumers, etc.) and/or for evaluating the content resources inthe network 100. The evaluation data store 308 also may includeinformation relating to evaluation processing tasks, for example, theidentification of users and user devices 106 that have received certaincontent resources or accessed certain applications, the status ofevaluations or evaluation histories for content resources, users, orapplications, and the like. Evaluation criteria may be stored in theevaluation data store 308 including data and/or instructions in the formof one or several electronic rubrics or scoring guides for use in theevaluation of the content, users, or applications. The evaluation datastore 308 also may include past evaluations and/or evaluation analysesfor users, content, and applications, including relative rankings,characterizations, explanations, and the like.

A model data store 309, also referred to herein as a model database 309can store information relating to one or several predictive models. Insome embodiments, these one or several predictive models can be used to:generate a prediction of the risk of a student-user not achieving one orseveral predetermined outcomes; generate a prediction of acategorization of the student-user, which categorization can indicate anexpected effect of one or several interventions on the student-user;and/or generate a prediction of a priority for any identifiedintervention.

In some embodiments, the risk model can comprise one or severalpredictive models based on, for example, one or several computerlearning techniques. In some embodiments, the risk model can be used togenerate a risk value for a student, which risk value characterizes therisk of the student-user not achieving the predetermined outcome suchas, for example, failing to complete a course or course of study,failing to graduate, failing to achieve a desired score or grade, or thelike. In some embodiments, the risk model can comprise, for example, adecision tree learning model. In some embodiments, the risk model cangenerate the risk value through the inputting of one or severalparameters, which parameters can be one or several values, into the riskmodel. These parameters can be generated based on one or severalfeatures or attributes of the student-user. The risk model, havingreceived the input parameters, can then generate the risk value.

In some embodiments, the categorization model can determine a categoryof the student-user. In some embodiments, the categorization model canbe used to generate one or several categorization values or identifiersthat identify a category of the student-user. In some embodiments, thiscategory can correspond to a likelihood of an intervention increasing ordecreasing the risk value. In some embodiments, the categories cancomprise a first category in which an intervention decreases the riskvalue, a second category in which an intervention increases the riskvalue, and a third category in which an intervention will not affect therisk value. In some embodiments, this third category can be furtherdivided into a first group in which the student-users will likely failto achieve the desired outcome regardless of intervention, and a secondgroup in which the student-users will likely achieve the desired outcomeregardless of intervention. In some embodiments, the categorizationmodel can determine the category of the student-user through the inputof one or several parameters relevant to the student-user into thecategorization model. In some embodiments, these parameters can begenerated from one or several features or attributes of the student-userthat can be, for example, extracted from data relating to thestudent-user.

In some embodiments, the priority model can determine a priority value,which can be a prediction of the importance of any determinedintervention. In some embodiments, this priority model can be determinedbased on information relating to the student-user for which the priorityvalue is determined. In some embodiments, this priority value can beimpacted by, for example, the value of the course associated with therisk value. In some embodiments, for example, the priority value mayindicate a lower priority for a risk in a non-essential course. In suchan embodiment, priority can be determined based on the credits of acourse, based on the relevance of a course to, for example, a degree ormajor, based on the role of the course as a pre-requisite to subsequentcourses, or the like.

A dashboard database 310 can include information for generating adashboard. In some embodiments, this information can identify one orseveral dashboard formats and/or architectures. As used herein, a formatrefers to how data is presented in a web page, and an architecturerefers to the data included in the web page and the format of that data.In some embodiments, the dashboard database 310 can comprise one orseveral pointers to other databases for retrieval of information forinclusion in the dashboard. Thus, in one embodiment, the dashboarddatabase 310 can comprise a pointer to all or portions of the userprofile database 301 to direct extraction of data from the user profiledatabase 301 for inclusion in the dashboard.

An intervention data source 311, also referred to herein as anintervention database can include information relating to one or severalinterventions, also referred to herein as one or several actions. Insome embodiments, this information can identify the one or severalinterventions, and how to implement the one or several interventions. Insome embodiments, these interventions can include, for example: acontact such as an email, a text, a telephone call, or an in-personvisit; a recommendation such as suggested supplemental material orsuggested involvement in a study group; a modification to enrollment orto the student-user schedule, or the like.

In some embodiments, the intervention database 311 can comprisedashboard data. In some embodiments, the dashboard data can include dataidentifying one or several alternate dashboard formats and/orarchitectures. In some embodiments, these one or several formats cancomprise the resizing and/or rearrangement of one or several items inthe dashboard (dashboard items), and the one or several architecturescan comprise the addition or subtraction of data from the dashboard andthe resizing and/or rearrangement of one or several items in thedashboard.

An error database 312 can include data relating to one or several errorsand/or anomalies arising during the communicating of content to and/orfrom a user device 106. In some embodiments, this communicating ofcontent can include, for example, providing of testing content and/orrelating to the providing of testing content. In some embodiments, theerror database can include a record of identified errors and/or receivederror signals, correspondence between the identified errors and/orreceived error signals, the number of corresponding errors, one orseveral triggering thresholds, one or several error solutions and/orremediations, or the like. In some embodiments, these errors canidentify one or more software errors, delays, crashes, network problems,or the like.

A location database 313 can include information identifying the locationof one or several of the user devices 106 used for the communicating ofcontent. In some embodiments, this location can be a geographiclocation, a political location such as, for example, identifying one orseveral of the user devices 106 as belonging to one of the end-usernetworks 107, or the like. The location information can be received fromthe one or several user devices 106 either prior to the start oftesting, or during testing.

A software database 314 can include information identifying hardwareand/or software used for the communicating of content. Specifically,this can identify the hardware and/or software, including the hardwareand/or software configuration for one or several of the user devices 106used for the communicating of content, and specifically used fortesting. This can include, for example, identification of an operationsystem, processor, modem, router, internet connection, antivirussoftware, security software, web browser, or the like. This informationcan be received from the one or more user devices either prior to thestart of testing, or during testing.

In addition to the illustrative databases described above, databaseserver(s) 104 (e.g., database servers, file-based storage servers, etc.)may include one or more external data aggregators 315. External dataaggregators 315 may include third-party data sources accessible to thecontent management network 100, but not maintained by the contentmanagement network 100. External data aggregators 315 may include anyelectronic information source relating to the users, content resources,or applications of the content distribution network 100. For example,external data aggregators 315 may be third-party databases containingdemographic data, education related data, consumer sales data, healthrelated data, and the like. Illustrative external data aggregators 315may include, for example, social networking web servers, public recordsdatabases, learning management systems, educational institution servers,business servers, consumer sales databases, medical record databases,etc. Data retrieved from various external data aggregators 315 may beused to verify and update user account information, suggest usercontent, and perform user and content evaluations.

With reference now to FIG. 4A, a block diagram is shown illustrating anembodiment of one or more content management servers 102 and/or errorservers 119 within a content distribution network 100. As discussedabove, content management server(s) 102 and/or error server(s) 119 mayinclude various server hardware and software components that manage thecontent resources within the content distribution network 100 andprovide interactive and adaptive content to users on various userdevices 106. For example, content management server(s) 102 and/or errorserver(s) 119 may provide instructions to and receive information fromthe other devices within the content distribution network 100, in orderto manage and transmit content resources, user data, and server orclient applications executing within the network 100.

A content management server 102 and/or error server 119 may include acontent customization system 402. The content customization system 402may be implemented using dedicated hardware within the contentdistribution network 100 (e.g., a content customization server 402), orusing designated hardware and software resources within a shared contentmanagement server 102 and/or error server 119. In some embodiments, thecontent customization system 402 may adjust the selection and adaptivecapabilities of content resources to match the needs and desires of theusers receiving the content. For example, the content customizationsystem 402 may query various databases and servers 104 to retrieve userinformation, such as user preferences and characteristics (e.g., from auser profile database 301), user access restrictions to contentrecourses (e.g., from a content access database 306), previous userresults and content evaluations (e.g., from an evaluation data store308), and/or the like. Based on the retrieved information from databases104 and other data sources, the content customization system 402 maymodify content resources for individual users.

In some embodiments, the content management system 402 can include arecommendation engine, also referred to herein as an adaptiverecommendation engine. In some embodiments, the recommendation enginecan select one or several pieces of content, also referred to herein asdata packets, for providing to a user. These data packets can beselected based on, for example, the information retrieved from thedatabase server 104 including, for example, the user profile database301, the content library database 303, the model database 309, or thelike. In one embodiment, for example, the recommendation engine canretrieve information from the user profile database 301 identifying, forexample, a skill level of the user. The recommendation engine canfurther retrieve information from the content library database 303identifying, for example, potential data packets for providing to theuser and the difficulty of those data packets and/or the skill levelassociated with those data packets.

The recommendation engine can use the evidence model to generate aprediction of the likelihood of one or several users providing a desiredresponse to some or all of the potential data packets. In someembodiments, the recommendation engine can pair one or several datapackets with selection criteria that may be used to determine whichpacket should be delivered to a student-user based on one or severalreceived responses from that student-user. In some embodiments, one orseveral data packets can be eliminated from the pool of potential datapackets if the prediction indicates either too high a likelihood of adesired response or too low a likelihood of a desired response. In someembodiments, the recommendation engine can then apply one or severalselection criteria to the remaining potential data packets to select adata packet for providing to the user. These one or several selectioncriteria can be based on, for example, criteria relating to a desiredestimated time for receipt of response to the data packet, one orseveral content parameters, one or several assignment parameters, or thelike.

A content management server 102 and/or error server 119 also may includea user management system 404. The user management system 404 may beimplemented using dedicated hardware within the content distributionnetwork 100 (e.g., a user management server 404), or using designatedhardware and software resources within a shared content managementserver 102 and/or error server 119. In some embodiments, the usermanagement system 404 may monitor the progress of users through varioustypes of content resources and groups, such as media compilations,courses or curriculums in training or educational contexts, interactivegaming environments, and the like. For example, the user managementsystem 404 may query one or more databases and/or datastore servers 104to retrieve user data such as associated content compilations orprograms, content completion status, user goals, results, and the like.

A content management server 102 and/or error server 119 also may includean evaluation system 406, also referred to herein as a responseprocessor. The evaluation system 406 may be implemented using dedicatedhardware within the content distribution network 100 (e.g., anevaluation server 406), or using designated hardware and softwareresources within a shared content management server 102. The evaluationsystem 406 may be configured to receive and analyze information fromuser devices 106 via, for example, the end-user server 105. For example,various ratings of content resources submitted by users may be compiledand analyzed, and then stored in a database (e.g., a content librarydatabase 303 and/or evaluation database 308) associated with thecontent. In some embodiments, the evaluation server 406 may analyze theinformation to determine the effectiveness or appropriateness of contentresources with, for example, a subject matter, an age group, a skilllevel, or the like. In some embodiments, the evaluation system 406 mayprovide updates to the content customization system 402 or the usermanagement system 404, with the attributes of one or more contentresources or groups of resources within the network 100. The evaluationsystem 406 also may receive and analyze user evaluation data from userdevices 106, supervisor devices 110, and administrator servers 116, etc.For instance, evaluation system 406 may receive, aggregate, and analyzeuser evaluation data for different types of users (e.g., end users,supervisors, administrators, etc.) in different contexts (e.g., mediaconsumer ratings, trainee or student comprehension levels, teachereffectiveness levels, gamer skill levels, etc.).

In some embodiments, the evaluation system 406 can be further configuredto receive one or several responses from the user and to determinewhether the one or several response are correct responses, also referredto herein as desired responses, or are incorrect responses, alsoreferred to herein as undesired responses. In some embodiments, one orseveral values can be generated by the evaluation system 406 to reflectuser performance in responding to the one or several data packets. Insome embodiments, these one or several values can comprise one orseveral scores for one or several responses and/or data packets.

A content management server 102 and/or error server 119 also may includea content delivery system 408. The content delivery system 408 may beimplemented using dedicated hardware within the content distributionnetwork 100 (e.g., a content delivery server 408), or using designatedhardware and software resources within a shared content managementserver 102 and/or error server 119. The content delivery system 408 caninclude a presentation engine that can be, for example, a softwaremodule running on the content delivery system.

The content delivery system 408, also referred to herein as thepresentation module or the presentation engine, may receive contentresources from the content customization system 402 and/or from the usermanagement system 404, and provide the resources to user devices 106.The content delivery system 408 may determine the appropriatepresentation format for the content resources based on the usercharacteristics and preferences, and/or the device capabilities of userdevices 106. If needed, the content delivery system 408 may convert thecontent resources to the appropriate presentation format and/or compressthe content before transmission. In some embodiments, the contentdelivery system 408 may also determine the appropriate transmissionmedia and communication protocols for transmission of the contentresources.

In some embodiments, the content delivery system 408 may includespecialized security and integration hardware 410, along withcorresponding software components to implement the appropriate securityfeatures content transmission and storage, to provide the supportednetwork and client access models, and to support the performance andscalability requirements of the network 100. The security andintegration layer 410 may include some or all of the security andintegration components 208 discussed above in FIG. 2, and may controlthe transmission of content resources and other data, as well as thereceipt of requests and content interactions, to and from the userdevices 106, supervisor devices 110, administrative servers 116, andother devices in the network 100.

With reference now to FIG. 4B, a flowchart illustrating one embodimentof a process 440 for data management is shown. In some embodiments, theprocess 440 can be performed by the content management server 102, andmore specifically by the content delivery system 408 and/or by thepresentation module or presentation engine. The process 440 begins atblock 442, wherein a data packet is identified. In some embodiments, thedata packet can be a data packet for providing to a student-user, andthe data packet can be identified by determining which data packet tonext provide to the user such as the student-user. In some embodiments,this determination can be performed by the content customization system402 and/or the recommendation engine.

After the data packet has been identified, the process 440 proceeds toblock 444, wherein the data packet is requested. In some embodiments,this can include the requesting of information relating to the datapacket such as the data forming the data packet. In some embodiments,this information can be requested from, for example, the content librarydatabase 303. After the data packet has been requested, the process 440proceeds to block 446, wherein the data packet is received. In someembodiments, the data packet can be received by the content deliverysystem 408 from, for example, the content library database 303.

After the data packet has been received, the process 440 proceeds toblock 448, wherein one or several data components are identified. Insome embodiments, for example, the data packet can include one orseveral data components which can, for example, contain different data.In some embodiments, one of these data components, referred to herein asa presentation component, can include content for providing to thestudent user, which content can include one or several requests and/orquestions and/or the like. In some embodiments, one of these datacomponents, referred to herein as a response component, can include dataused in evaluating one or several responses received from the userdevice 106 in response to the data packet, and specifically in responseto the presentation component and/or the one or several requests and/orquestions of the presentation component. Thus, in some embodiments, theresponse component of the data packet can be used to ascertain whetherthe user has provided a desired response or an undesired response.

After the data components have been identified, the process 440 proceedsto block 450, wherein a delivery data packet is identified. In someembodiments, the delivery data packet can include the one or severaldata components of the data packets for delivery to a user such as thestudent-user via the user device 106. In some embodiments, the deliverypacket can include the presentation component, and in some embodiments,the delivery packet can exclude the response packet. After the deliverydata packet has been generated, the process 440 proceeds to block 452,wherein the delivery data packet is presented to the user device 106. Insome embodiments, this can include providing the delivery data packet tothe user device 106 via, for example, the communication network 120.

After the delivery data packet has been provided to the user device, theprocess 440 proceeds to block 454, wherein the data packet and/or one orseveral components thereof is sent to and/or provided to the responseprocessor. In some embodiments, this sending of the data packet and/orone or several components thereof to the response processor can includereceiving a response from the student-user, and sending the response tothe student-user to the response processor simultaneous with the sendingof the data packet and/or one or several components thereof to theresponse processor. In some embodiments, for example, this can includeproviding the response component to the response processor. In someembodiments, the response component can be provided to the responseprocessor from the content delivery system 408.

With reference now to FIG. 4C, a flowchart illustrating one embodimentof a process 460 for evaluating a response is shown. In someembodiments, the process can be performed by the evaluation system 406.In some embodiments, the process 460 can be performed by the evaluationsystem 406 in response to the receipt of a response from the user device106.

The process 460 begins at block 462, wherein a response is receivedfrom, for example, the user device 106 via, for example, thecommunication network 120. After the response has been received, theprocess 460 proceeds to block 464, wherein the data packet associatedwith the response is received. In some embodiments, this can includereceiving all or one or several components of the data packet such as,for example, the response component of the data packet. In someembodiments, the data packet can be received by the response processorfrom the presentation engine.

After the data packet has been received, the process 460 proceeds toblock 466, wherein the response type is identified. In some embodiments,this identification can be performed based on data, such as metadataassociated with the response. In other embodiments, this identificationcan be performed based on data packet information such as the responsecomponent.

In some embodiments, the response type can identify one or severalattributes of the one or several requests and/or questions of the datapacket such as, for example, the request and/or question type. In someembodiments, this can include identifying some or all of the one orseveral requests and/or questions as true/false, multiple choice, shortanswer, essay, or the like.

After the response type has been identified, the process 460 proceeds toblock 468, wherein the data packet and the response are compared todetermine whether the response comprises a desired response and/or anundesired response. In some embodiments, this can include comparing thereceived response and the data packet to determine if the receivedresponse matches all or portions of the response component of the datapacket, to determine the degree to which the received response matchesall or portions of the response component, to determine the degree towhich the receive response embodies one or several qualities identifiedin the response component of the data packet, or the like. In someembodiments, this can include classifying the response according to oneor several rules. In some embodiments, these rules can be used toclassify the response as either desired or undesired. In someembodiments, these rules can be used to identify one or several errorsand/or misconceptions evidenced in the response. In some embodiments,this can include, for example: use of natural language processingsoftware and/or algorithms; use of one or several digital thesauruses;use of lemmatization software, dictionaries, and/or algorithms; or thelike.

After the data packet and the response have been compared, the process460 proceeds to block 470 wherein response desirability is determined.In some embodiments this can include, based on the result of thecomparison of the data packet and the response, whether the response isa desired response or is an undesired response. In some embodiments,this can further include quantifying the degree to which the response isa desired response. This determination can include, for example,determining if the response is a correct response, an incorrectresponse, a partially correct response, or the like. In someembodiments, the determination of response desirability can include thegeneration of a value characterizing the response desirability and thestoring of this value in one of the databases 104 such as, for example,the user profile database 301. After the response desirability has beendetermined, the process 460 proceeds to block 472, wherein an assessmentvalue is generated. In some embodiments, the assessment value can be anaggregate value characterizing response desirability for one or more aplurality of responses. This assessment value can be stored in one ofthe databases 104 such as the user profile database 301.

With reference now to FIG. 5, a block diagram of an illustrativecomputer system is shown. The system 500 may correspond to any of thecomputing devices or servers of the content distribution network 100described above, or any other computing devices described herein, andspecifically can include, for example, one or several of the userdevices 106, the supervisor device 110, and/or any of the servers 102,104, 108, 112, 114, 116. In this example, computer system 500 includesprocessing units 504 that communicate with a number of peripheralsubsystems via a bus subsystem 502. These peripheral subsystems include,for example, a storage subsystem 510, an I/O subsystem 526, and acommunications subsystem 532.

Bus subsystem 502 provides a mechanism for letting the variouscomponents and subsystems of computer system 500 communicate with eachother as intended. Although bus subsystem 502 is shown schematically asa single bus, alternative embodiments of the bus subsystem may utilizemultiple buses. Bus subsystem 502 may be any of several types of busstructures including a memory bus or memory controller, a peripheralbus, and a local bus using any of a variety of bus architectures. Sucharchitectures may include, for example, an Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Association (VESA) localbus, and Peripheral Component Interconnect (PCI) bus, which can beimplemented as a Mezzanine bus manufactured to the IEEE P1386.1standard.

Processing unit 504, which may be implemented as one or more integratedcircuits (e.g., a conventional microprocessor or microcontroller),controls the operation of computer system 500. One or more processors,including single core and/or multicore processors, may be included inprocessing unit 504. As shown in the figure, processing unit 504 may beimplemented as one or more independent processing units 506 and/or 508with single or multicore processors and processor caches included ineach processing unit. In other embodiments, processing unit 504 may alsobe implemented as a quad-core processing unit or larger multicoredesigns (e.g., hexa-core processors, octo-core processors, ten-coreprocessors, or greater.

Processing unit 504 may execute a variety of software processes embodiedin program code, and may maintain multiple concurrently executingprograms or processes. At any given time, some or all of the programcode to be executed can be resident in processor(s) 504 and/or instorage subsystem 510. In some embodiments, computer system 500 mayinclude one or more specialized processors, such as digital signalprocessors (DSPs), outboard processors, graphics processors,application-specific processors, and/or the like.

I/O subsystem 526 may include device controllers 528 for one or moreuser interface input devices and/or user interface output devices 530.User interface input and output devices 530 may be integral with thecomputer system 500 (e.g., integrated audio/video systems, and/ortouchscreen displays), or may be separate peripheral devices which areattachable/detachable from the computer system 500. The I/O subsystem526 may provide one or several outputs to a user by converting one orseveral electrical signals to user perceptible and/or interpretableform, and may receive one or several inputs from the user by generatingone or several electrical signals based on one or several user-causedinteractions with the I/O subsystem such as the depressing of a key orbutton, the moving of a mouse, the interaction with a touchscreen ortrackpad, the interaction of a sound wave with a microphone, or thelike.

Input devices 530 may include a keyboard, pointing devices such as amouse or trackball, a touchpad or touch screen incorporated into adisplay, a scroll wheel, a click wheel, a dial, a button, a switch, akeypad, audio input devices with voice command recognition systems,microphones, and other types of input devices. Input devices 530 mayalso include three dimensional (3D) mice, joysticks or pointing sticks,gamepads and graphic tablets, and audio/visual devices such as speakers,digital cameras, digital camcorders, portable media players, webcams,image scanners, fingerprint scanners, barcode reader 3D scanners, 3Dprinters, laser rangefinders, and eye gaze tracking devices. Additionalinput devices 530 may include, for example, motion sensing and/orgesture recognition devices that enable users to control and interactwith an input device through a natural user interface using gestures andspoken commands, eye gesture recognition devices that detect eyeactivity from users and transform the eye gestures as input into aninput device, voice recognition sensing devices that enable users tointeract with voice recognition systems through voice commands, medicalimaging input devices, MIDI keyboards, digital musical instruments, andthe like.

Output devices 530 may include one or more display subsystems, indicatorlights, or non-visual displays such as audio output devices, etc.Display subsystems may include, for example, cathode ray tube (CRT)displays, flat-panel devices, such as those using a liquid crystaldisplay (LCD) or plasma display, light-emitting diode (LED) displays,projection devices, touch screens, and the like. In general, use of theterm “output device” is intended to include all possible types ofdevices and mechanisms for outputting information from computer system500 to a user or other computer. For example, output devices 530 mayinclude, without limitation, a variety of display devices that visuallyconvey text, graphics and audio/video information such as monitors,printers, speakers, headphones, automotive navigation systems, plotters,voice output devices, and modems.

Computer system 500 may comprise one or more storage subsystems 510,comprising hardware and software components used for storing data andprogram instructions, such as system memory 518 and computer-readablestorage media 516. The system memory 518 and/or computer-readablestorage media 516 may store program instructions that are loadable andexecutable on processing units 504, as well as data generated during theexecution of these programs.

Depending on the configuration and type of computer system 500, systemmemory 318 may be stored in volatile memory (such as random accessmemory (RAM) 512) and/or in non-volatile storage drives 514 (such asread-only memory (ROM), flash memory, etc.) The RAM 512 may contain dataand/or program modules that are immediately accessible to and/orpresently being operated and executed by processing units 504. In someimplementations, system memory 518 may include multiple different typesof memory, such as static random access memory (SRAM) or dynamic randomaccess memory (DRAM). In some implementations, a basic input/outputsystem (BIOS), containing the basic routines that help to transferinformation between elements within computer system 500, such as duringstart-up, may typically be stored in the non-volatile storage drives514. By way of example, and not limitation, system memory 518 mayinclude application programs 520, such as client applications, Webbrowsers, mid-tier applications, server applications, etc., program data522, and an operating system 524.

Storage subsystem 510 also may provide one or more tangiblecomputer-readable storage media 516 for storing the basic programmingand data constructs that provide the functionality of some embodiments.Software (programs, code modules, instructions) that when executed by aprocessor provide the functionality described herein may be stored instorage subsystem 510. These software modules or instructions may beexecuted by processing units 504. Storage subsystem 510 may also providea repository for storing data used in accordance with the presentinvention.

Storage subsystem 300 may also include a computer-readable storage mediareader that can further be connected to computer-readable storage media516. Together and, optionally, in combination with system memory 518,computer-readable storage media 516 may comprehensively representremote, local, fixed, and/or removable storage devices plus storagemedia for temporarily and/or more permanently containing, storing,transmitting, and retrieving computer-readable information.

Computer-readable storage media 516 containing program code, or portionsof program code, may include any appropriate media known or used in theart, including storage media and communication media, such as but notlimited to, volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information. This can include tangible computer-readable storagemedia such as RAM, ROM, electronically erasable programmable ROM(EEPROM), flash memory or other memory technology, CD-ROM, digitalversatile disk (DVD), or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or other tangible computer readable media. This can also includenontangible computer-readable media, such as data signals, datatransmissions, or any other medium which can be used to transmit thedesired information and which can be accessed by computer system 500.

By way of example, computer-readable storage media 516 may include ahard disk drive that reads from or writes to non-removable, nonvolatilemagnetic media, a magnetic disk drive that reads from or writes to aremovable, nonvolatile magnetic disk, and an optical disk drive thatreads from or writes to a removable, nonvolatile optical disk such as aCD ROM, DVD, and Blu-Ray® disk, or other optical media.Computer-readable storage media 516 may include, but is not limited to,Zip® drives, flash memory cards, universal serial bus (USB) flashdrives, secure digital (SD) cards, DVD disks, digital video tape, andthe like. Computer-readable storage media 516 may also include,solid-state drives (SSD) based on non-volatile memory such asflash-memory based SSDs, enterprise flash drives, solid state ROM, andthe like, SSDs based on volatile memory such as solid state RAM, dynamicRAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, andhybrid SSDs that use a combination of DRAM and flash memory based SSDs.The disk drives and their associated computer-readable media may providenon-volatile storage of computer-readable instructions, data structures,program modules, and other data for computer system 500.

Communications subsystem 532 may provide a communication interface fromcomputer system 500 and external computing devices via one or morecommunication networks, including local area networks (LANs), wide areanetworks (WANs) (e.g., the Internet), and various wirelesstelecommunications networks. As illustrated in FIG. 5, thecommunications subsystem 532 may include, for example, one or morenetwork interface controllers (NICs) 534, such as Ethernet cards,Asynchronous Transfer Mode NICs, Token Ring NICs, and the like, as wellas one or more wireless communications interfaces 536, such as wirelessnetwork interface controllers (WNICs), wireless network adapters, andthe like. As illustrated in FIG. 5, the communications subsystem 532 mayinclude, for example, one or more location determining features 538 suchas one or several navigation system features and/or receivers, and thelike. Additionally and/or alternatively, the communications subsystem532 may include one or more modems (telephone, satellite, cable, ISDN),synchronous or asynchronous digital subscriber line (DSL) units,FireWire® interfaces, USB® interfaces, and the like. Communicationssubsystem 536 also may include radio frequency (RF) transceivercomponents for accessing wireless voice and/or data networks (e.g.,using cellular telephone technology, advanced data network technology,such as 3G, 4G or EDGE (enhanced data rates for global evolution), WiFi(IEEE 802.11 family standards, or other mobile communicationtechnologies, or any combination thereof), global positioning system(GPS) receiver components, and/or other components.

The various physical components of the communications subsystem 532 maybe detachable components coupled to the computer system 500 via acomputer network, a FireWire® bus, or the like, and/or may be physicallyintegrated onto a motherboard of the computer system 500. Communicationssubsystem 532 also may be implemented in whole or in part by software.

In some embodiments, communications subsystem 532 may also receive inputcommunication in the form of structured and/or unstructured data feeds,event streams, event updates, and the like, on behalf of one or moreusers who may use or access computer system 500. For example,communications subsystem 532 may be configured to receive data feeds inreal-time from users of social networks and/or other communicationservices, web feeds such as Rich Site Summary (RSS) feeds, and/orreal-time updates from one or more third party information sources(e.g., data aggregators 315). Additionally, communications subsystem 532may be configured to receive data in the form of continuous datastreams, which may include event streams of real-time events and/orevent updates (e.g., sensor data applications, financial tickers,network performance measuring tools, clickstream analysis tools,automobile traffic monitoring, etc.). Communications subsystem 532 mayoutput such structured and/or unstructured data feeds, event streams,event updates, and the like to one or more databases 104 that may be incommunication with one or more streaming data source computers coupledto computer system 500.

Due to the ever-changing nature of computers and networks, thedescription of computer system 500 depicted in the figure is intendedonly as a specific example. Many other configurations having more orfewer components than the system depicted in the figure are possible.For example, customized hardware might also be used and/or particularelements might be implemented in hardware, firmware, software, or acombination. Further, connection to other computing devices, such asnetwork input/output devices, may be employed. Based on the disclosureand teachings provided herein, a person of ordinary skill in the artwill appreciate other ways and/or methods to implement the variousembodiments.

With reference now to FIG. 6, a block diagram illustrating oneembodiment of the communication network is shown. Specifically, FIG. 6depicts one hardware configuration in which messages are exchangedbetween a source hub 602 via the communication network 120 that caninclude one or several intermediate hubs 604. In some embodiments, thesource hub 602 can be any one or several components of the contentdistribution network generating and initiating the sending of a message,and the terminal hub 606 can be any one or several components of thecontent distribution network 100 receiving and not re-sending themessage. In some embodiments, for example, the source hub 602 can be oneor several of the user device 106, the supervisor device 110, and/or theserver 102, and the terminal hub 606 can likewise be one or several ofthe user device 106, the supervisor device 110, and/or the server 102.In some embodiments, the intermediate hubs 604 can include any computingdevice that receives the message and resends the message to a next node.

As seen in FIG. 6, in some embodiments, each of the hubs 602, 604, 606can be communicatingly connected with the data store 104. In such anembodiments, some or all of the hubs 602, 604, 606 can send informationto the data store 104 identifying a received message and/or any sent orresent message. This information can, in some embodiments, be used todetermine the completeness of any sent and/or received messages and/orto verify the accuracy and completeness of any message received by theterminal hub 606.

In some embodiments, the communication network 120 can be formed by theintermediate hubs 604. In some embodiments, the communication network120 can comprise a single intermediate hub 604, and in some embodiments,the communication network 120 can comprise a plurality of intermediatehubs. In one embodiment, for example, and as depicted in FIG. 6, thecommunication network 120 includes a first intermediate hub 604-A and asecond intermediate hub 604-B.

With reference now to FIG. 7, a block diagram illustrating oneembodiment of user device 106 and supervisor device 110 communication isshown. In some embodiments, for example, a user may have multipledevices that can connect with the content distribution network 100 tosend or receive information. In some embodiments, for example, a usermay have a personal device such as a mobile device, a Smartphone, atablet, a Smartwatch, a laptop, a PC, or the like. In some embodiments,the other device can be any computing device in addition to the personaldevice. This other device can include, for example, a laptop, a PC, aSmartphone, a tablet, a Smartwatch, or the like. In some embodiments,the other device differs from the personal device in that the personaldevice is registered as such within the content distribution network 100and the other device is not registered as a personal device within thecontent distribution network 100.

Specifically with respect to FIG. 7, the user device 106 can include apersonal user device 106-A and one or several other user devices 106-B.In some embodiments, one or both of the personal user device 106-A andthe one or several other user devices 106-B can be communicatinglyconnected to the content management server 102 and/or to the system 122.Similarly, the supervisor device 110 can include a personal supervisordevice 110-A and one or several other supervisor devices 110-B. In someembodiments, one or both of the personal supervisor device 110-A and theone or several other supervisor devices 110-B can be communicatinglyconnected to the content management server 102 and/or to the navigationsystem 124.

In some embodiments, the content distribution network can send one ormore alerts to one or more user devices 106 and/or one or moresupervisor devices 110 via, for example, the communication network 120.In some embodiments, the receipt of the alert can result in thelaunching of an application within the receiving device, and in someembodiments, the alert can include a link that, when selected, launchesthe application or navigates a web-browser of the device of the selectorof the link to page or portal associated with the alert.

In some embodiments, for example, the providing of this alert caninclude the identification of one or several user devices 106 and/orstudent-user accounts associated with the student-user and/or one orseveral supervisor devices 110 and/or supervisor-user accountsassociated with the supervisor-user. After these one or several devices106, 110 and/or accounts have been identified, the providing of thisalert can include determining an active device of the devices 106, 110based on determining which of the devices 106, 110 and/or accounts areactively being used, and then providing the alert to that active device.

Specifically, if the user is actively using one of the devices 106, 110such as the other user device 106-B and the other supervisor device110-B, and/or accounts, the alert can be provided to the user via thatother device 106-B, 110-B and/or account that is actively being used. Ifthe user is not actively using an other device 106-B, 110-B and/oraccount, a personal device 106-A, 110-A device, such as a smart phone ortablet, can be identified and the alert can be provided to this personaldevice 106-A, 110-A. In some embodiments, the alert can include code todirect the default device to provide an indicator of the received alertsuch as, for example, an aural, tactile, or visual indicator of receiptof the alert.

In some embodiments, the recipient device 106, 110 of the alert canprovide an indication of receipt of the alert. In some embodiments, thepresentation of the alert can include the control of the I/O subsystem526 to, for example, provide an aural, tactile, and/or visual indicatorof the alert and/or of the receipt of the alert. In some embodiments,this can include controlling a screen of the supervisor device 110 todisplay the alert, data contained in alert and/or an indicator of thealert.

With reference now to FIG. 8, a block diagram illustrating oneembodiment of the connection of user devices 106 to a supervisor device110 is shown. In some embodiments, one or several of the user devices106 can be connected to a supervisor device 110 in a classroomenvironment and/or to form a virtual classroom. In embodiments in whichthe devices 106, 110 are connected to form a virtual classroom, thedevices can be connected via, for example, a WAN, a cellular network, atelephone communication network, or the like.

In embodiments in which the devices 106, 110 are connected in aclassroom environment. In such a classroom environment, the user devices106 and the supervisor device 110 can be connected to each other via,for example, a Local Area Network (LAN). This configuration canfacilitate the quick transfer of data between the devices 106, 110 andcan increase the speed with which survey data can be provided to theuser devices 106 and survey data can be received form the user crevices106 and provided to the supervisor device 110. In some such embodiments,the supervisor device 110 can be further connected with the back-endcomponents 122 and can serve as a conduit for survey data from the userdevices 106 to the back-end components 122. In such an embodiment, thesupervisor device 110 can receive survey data from the user devices 106,can identify some or all of the survey data for local analysis, and canprovide all of the survey data or the data not identified for localanalysis to the back-end components 122. The supervisor device 110 canadditionally, in some embodiments, locally analyze the portion of thesurvey data identified for local analysis and can use the analysis ofthis portion of the survey data to generate and provide one or morerecommendations relating to content being delivered to the users of theuser devices 106.

With reference now to FIG. 9, a block diagram of one embodiment of theend-user server 105 and/or the user device 106 is shown. As discussedabove, the end-user server 105 and/or the user device 106 can beconfigured to provide information to and/or receive information fromother components of the content delivery network 100. The end-userserver 105 and/or the user device can 106 access the content deliverynetwork 100 through any desired means or technology, including, forexample, a webpage such as, for example, a social network service page,or a web portal. As depicted in FIG. 9, the end-user server 105 and/orthe user device 106 can include a network interface 600. The networkinterface 600 allows the end-user server 105 and/or user device 106 toaccess the other components of the content delivery network 100. Thenetwork interface 600 can include features configured to send andreceive information, including, for example, an antenna, a modem, atransmitter, receiver, or any other feature that can send and receiveinformation. The network interface 600 can communicate via telephone,cable, fiber-optic, or any other wired communication network. In someembodiments, the network interface 600 can communicate via cellularnetworks, WLAN networks, or any other wireless network.

The end-user server 105 and/or the user device 106 can include, forexample, an error detection engine 601. The error detection engine 601can be configured to detect an error in the presentation of testingcontent to a user. In some embodiments, an error, also referred toherein as an anomaly or a detrimental anomaly, can include, for example,a delay in providing testing content, a stoppage of the delivery of thetesting content, the performing of an unauthorized operation by the userdevice 106, a loss of network connectivity, a firewall problem, asecurity and/or authorization problem, or the like. In some embodiments,the error detection engine 601 can comprise thresholds indicative ofnormal operation and can compare current operation levels to thosethresholds to identify one or several errors. Alternatively, errors canbe detected in any other known manner.

The end-user server 105 and/or the user device 106 can include a userinterface 605 that communicates information to, and receives inputsfrom, a user. The user interface 605 can include a screen, a speaker, amonitor, a keyboard, a microphone, a mouse, a touchpad, a keypad, atouchscreen, or any other feature or features that can receive inputsfrom a user and provide information to a user.

With reference now to FIGS. 10 and 11, a flowchart illustrating oneembodiment of a process 700 for automatic electronic communication errordetection and remediation is shown. In some embodiments, automaticelectronic communication error detection and remediation can includeautomatic testing error detection and remediation. In some embodiments,automatic electronic communication error detection and remediation canbe performed to automatically identify and/or remedy errors and/orevents that interfere with the delivery of content to a user device 106and/or from the user device 106. In some embodiments, this can include,for example, one or several software, hardware, and/or network errors orevents. In some embodiments in which the electronic communicationcomprises a test, these errors can prevent and/or interfere withdelivery and/or communication of one or several test data packets,prevent and/or interfere with response to one or several test datapackets, or the like. In some embodiments, this can degrade the qualityof electronic communication and/or affect the reliability of the test inidentifying a skill level.

The process 1000 can begin at block 1002, wherein an initiationindicator is received. In some embodiments, this indicator can indicatethe start of electronic communication, and specifically can indicate thestart of testing. In some embodiments, the initiation indicator cancomprise an electronic message that can, for example, comprise one orseveral values. In some embodiments, this indication can be received bythe error server 119 in response to the providing of testing content tothe student, which testing content was received by the user device 106from the content management server 102. After the indication of theinitiation of electronic communication has been received, the process1000 proceeds to block 1004, wherein location data, also referred toherein as location information is received. In some embodiments, thislocation information can be received from the user device 106 and canidentify, for example, the geographic location of the user device, theend-user network 107 to which the user device 106 belongs, or the like.In some embodiments, this location information can be generated by theuser device 106 via, for example, the navigation system 124, and in someembodiments, this location information can be stored in memory of theuser device 106. The location information can be received via anelectronic signal at the error server 119.

After the location information has been received, the process 1000proceeds to block 1006, wherein the hardware information is received. Insome embodiments, the hardware information can be received in the formof an electronic signal by the error server 119 from the user device106. The hardware information can identify the hardware of the userdevice 106, and/or of the end-user network 107 to which the user device106 belongs. In some embodiments, the hardware information furtheridentifies any hardware unique to the communication network 120 viawhich the end-user network 107 connects to the content network 122.

After the hardware information has been received, the process 1000proceeds to block 1008, wherein software information is received. Insome embodiments, the software information can identify the softwarerunning on the user device 106 and/or on the end-user server 105 of theend-user network 107. This software can include, for example, theoperating system software, the software delivering the testing content,any security and/or antivirus software, web browsing software, or thelike. The software information can be received by the error server 119from the user device 106 via an electrical signal.

After the software information has been received, the process 1000proceeds to block 1010, wherein configuration information is received.In some embodiments, the configuration information can identify one orseveral configurations of software and/or hardware of the user device106 or of the other components of the end-user network 107. Theconfiguration information can include information relating to, forexample, one or several firewall settings, one or several security levelsettings, one or several modes of operation such as, for example, screenresolution and/or color resolution levels, memory allocation to RAM, orthe like.

After the configuration information has been received, the process 1000proceeds to block 1012, wherein an error message, also referred toherein as a feedback communication, is received. The error message canspecify one or several errors occurring during the electroniccommunication, and specifically during presentation of testing contentand/or during the testing. In some embodiments, the error message can begenerated by the user device 106, and specifically can be generated bythe error detection system 601 of the user device 106 and can beprovided to the error server 119 in the form of an electrical signal.The error message can include data indicative of the error. This datacan identify, for example, an error code, and error type, an effect ofthe error, potential sources of the error including, for example, thestarting on a non-approved software, or the like.

After the error message is received, the process 1000 proceeds to block1014, wherein the error message is compared to the error data. In someembodiments, this step can include retrieving error data from the errordatabase 312, and then identifying any correspondences between the errormessage and the error data. After the error message has been comparedwith the error data, the process 1000 proceeds to decision state 1016,located on FIG. 11, wherein it is determined whether the at least one ofthe one or more errors identified in the error message is the same typeof error as identified in the error data. In some embodiments, this caninclude determining whether any correspondence was identified betweenthe at least one or more errors identified in the error message and theerrors identified in the error data. Specifically, this can be performedto determine whether one or several errors and/or types of errors arecurring and/or the frequency of the recurrence of these one or severalerrors.

If it is determined that the same type of error was identified in theerror message and in the error data, then the process proceeds to block1018, wherein a count associated with the identified error type isincremented. In some embodiments, for example, this count can track thenumber of corresponding and/or matching errors. This count can be storedin one of the databases 104, and can be specifically stored in the errordatabase 312.

After the count has been incremented, or returning again to decisionstate 1016, if it is determined that the errors in the error message andin the error data are not the same type, then the process 1000 proceedsto identify the existence of any error correlations. In someembodiments, this can include determining if the errors are associatedwith, for example, one or several attributes such as: location of theuser devices 106 experiencing the errors; the type of electroniccommunication and/or electronic content associated with the error; thetest associated with the error; the question type and/or response typeassociated with the error; the code in which the content is written;embedded code within the content; a test form; software and/or hardwarerequirements of the user device 106 associated with the error; processeson the user device 106 at the time of the error; the question and/orresponse associated with the error; one or several of the date, day ofthe week, and/or time of the error; or the like. In some embodiments,this can include, for example, identifying commonalities in attributesassociated with the errors and/or error data and determining thestatistical significance of these commonalities. In some embodiments,this statistical significance can be measured in the form of acomparison of a threshold to a count of the number of errors associatedwith the attribute, and in some embodiments, this can include one orseveral measures of statistical significance that normalize for thesample size, such as, for example, a percent of users sharing a commonattribute that experience an error. One example of steps for determiningan error correlation is shown in FIG. 11 with respect to 1020-1028. Inthese steps, the common attribute that is evaluated is a location,although it will be appreciated that any of the above-mentionedattributes can be likewise evaluated.

After the count has been incremented, or returning again to decisionstate 1016, if it is determined that the errors in the error message andin the error data are not the same type, then the process 1000 proceedsto decision state 1020, wherein it is determined if the errors of theerror message and the error data occurred in the same location. In someembodiments, this can include determining whether multiple errorsoccurred within a predetermined distance of one another, oralternatively, determining if multiple errors occurred within a singleend-user network 107, occurred with a single connectivity providerand/or with a single communications network provider, or the like.

If it is determined that the errors of the error message and the errordata did not occur in the same location, the process 1000 proceeds toblock 1022, wherein the error data is updated with information relatingto the one or several errors identified in the error message. In someembodiments, the error data can be updated in one of the databases 104,and particularly in the error database 312.

After the error data has been updated, or returning again to decisionstate 1020, if it is determined that the errors in the error message andin the error data occurred occur at the same location, the process 1000proceeds to block 1024, wherein a location count associated with theidentified error location is incremented. In some embodiments, forexample, this location count can track the number of correspondingand/or matching errors, and specifically the number of errors occurringat the same location. This location count can be stored in one of thedatabases 104, and can be specifically stored in the error database 312

After the location count has been incremented, or returning to block1022, after the error data is updated, the process 1000 proceeds toblock 1026, wherein the counts, and specifically, wherein the type countand/or the location count are compared to the location and/or typecounts. This can include determining whether the value for the count islarger than threshold values. After the count has been compared to thethreshold values, the process 1000 proceeds to decision state 1028wherein it is determined if the count exceeds the threshold value. Ifthe count does not exceed the threshold value, then the process 1000proceeds to block 1030, and can terminate, or alternatively, can waituntil the receipt of a next or additional error message, at which pointthe process 1000 can return to block 1012 and proceed as outlined above.

Returning again to decision state 1028, if it is determined that thecount exceeds the threshold value, then the process 1000 proceeds toblock 1032, wherein an error warning and/or alert is generated and/orsent. In some embodiments, the error warning can comprise a messageidentifying one or several key aspects of the corresponding errorsincluding, for example, one or several testers affected by the error,one or several tests affected by the error, one or several testquestions affected by the error, or the like. In some embodiments, theerror alert can be generated and/or sent by one of the servers of thecontent distribution network 100 such as, for example, one of theend-user servers 105-A, 105-B, the error server 119, and/or the server102. In some embodiments,

In some embodiments, one or more alerts can be sent to one or more userdevices 106 and/or one or more supervisor devices 110 via, for example,the communication network 120. In some embodiments, the receipt of thealert can result in the launching of an application within the receivingdevice, and in some embodiments, the alert can include a link that, whenselected, launches the application or navigates a web-browser of thedevice of the selector of the link to page or portal associated with thealert. In some embodiments, for example, the providing of this alert caninclude the identification of one or several user devices 106 and/orstudent-user accounts associated with the student-user and/or one orseveral supervisor devices 110 and/or supervisor-user accountsassociated with the supervisor-user. After these one or several devices106, 110 and/or accounts have been identified, the providing of thisalert can include determining an active device of the devices 106, 110based on determining which of the devices 106, 110 and/or accounts areactively being used, and then providing the alert to that active device.In some embodiments, the alert can comprise data identifying one orseveral errors and/or one or several attributes associated with theerrors.

Specifically, if the user is actively using one of the devices 106, 110such as the other user device 106-B and the other supervisor device110-B, and/or accounts, the alert can be provided to the user via thatother device 106-B, 110-B and/or account that is actively being used. Ifthe user is not actively using an other device 106-B, 110-B and/oraccount, a personal device 106-A, 110-A device, such as a smart phone ortablet, can be identified and the alert can be provided to this personaldevice 106-A, 110-A. In some embodiments, the alert can include code todirect the default device to provide an indicator of the received alertsuch as, for example, an aural, tactile, or visual indicator of receiptof the alert.

In some embodiments, the recipient device 106, 110 of the alert canprovide an indication of receipt of the alert. In some embodiments, thepresentation of the alert can include the control of the I/O subsystem526 to, for example, provide an aural, tactile, and/or visual indicatorof the alert and/or of the receipt of the alert. In some embodiments,this can include controlling a screen of the supervisor device 110 todisplay the alert, data contained in alert and/or an indicator of thealert.

After the error warning has been generated, the process 1000 proceeds toblock 1034, wherein a remedy is selected. In some embodiments, thisremedy, also referred to herein as a proposed remedy, can be a solution,or a suggested solution to the underlying cause of the error. This caninclude, for example, a restarting of affected user devices 106, are-establishment of network connections, a reconfiguration of softwareor hardware, a removal of hardware or software, or the like. This remedycan be selected based on information in the error message and based onremedy information that can be stored in the error database.

After the remedy has been selected, the process 1000 proceeds to block1036, wherein the remedy is provided to a user. In some embodiments, theuser recipient of the remedy can be the tester using an affected userdevice 106, a system administrator, or the like. In some embodiments inwhich the tester using the affected user device 106 is the userrecipient, the remedy can be provided via the tester's user device 106.In some embodiments, the remedy can be provided to the user as part ofthe error alert, and in some embodiments, the remedy can be provided tothe user separate from the error alert.

With reference now to FIG. 12, a flowchart illustrating one embodimentof a process 1200 for generating a performance adjustment based on adetected error, such as a detected testing error, is shown. In someembodiments, this process 1200 can be performed to gather informationrelating to a user's performance and how one or several errors affectedthe user's performance. In some embodiments, this process 1200 can befurther used in, for example, in low-stakes testing.

The process 1200 begins at block 1202, wherein content such as, forexample, a test, a question, a prompt, or the like is received. In someembodiments, this can include the receipt of testing content, and thestoring of the testing content in one of the databases 104 such as, forexample, the content library database 303.

After the test has been received, the process 1200 proceeds to block1204, wherein log-in information for a plurality of users is received.In some embodiments, this information can be received as part of theinitiation of testing, and the log-in information can be received by,for example, the content management server 102 from one or several ofthe user devices 106.

After the log-in information has been received, the process 1200proceeds to block 1206, wherein the content is provided to some or allof the plurality of users. In some embodiments, this can include theretrieval of the content from one of the databases 104, and providingthe content to the user device(s) 106 being used by the some or all ofthe plurality of users.

After the content has been provided to the plurality of users, theprocess 1200 proceeds to block 1208, wherein a feedback connection isestablished. In some embodiments, this can include establishing acommunication connection between one or several of the user devices 106and the error server 119, that is separate from the communicationconnection via which the content was provided to the user devices 106and in some embodiments, the feedback connection can be established viathe communication connection through which the content was provided tothe user devices 106.

After the feedback connection has been established, the process 1200proceeds to block 1210, wherein a feedback communication is received.The feedback communication can be received by the error server 119.After the feedback communication has been received, the process 1200proceeds to block 1212, wherein user result data such as, for example,test result data is received. The user result data can include dataidentifying, for example, a score achieved by the student-user, anidentification of test questions that were correctly answered, anidentification of test questions that were incorrectly answered,information relating to the time taken for completion of the test and/orfor responding to some or all of the test questions.

After the user result data has been received, the process 1200 proceedsto block 1214, wherein the historical performance data is retrieved. Insome embodiments, this historical data can be relevant to the currenttesters and/or to previous testers. In some embodiments, this data caninclude data relating to previous test results of current users. In someembodiments, this data can include information relating to previous userscores and/or how the previous users' scores were affected by one orseveral errors occurring during the test. This data can be retrievedfrom the database 104, and specifically can be retrieved from the userprofile database 301.

After the historical performance data has been retrieved, the process1200 proceeds to block 1216, wherein one or several student experiencecohorts are identified. In some embodiments, a student experience cohortcan comprise a group of students who experienced a similar error typeduring an electronic communication. In some embodiments, a studentexperience cohort can be further defined by the subject matter of theelectronic communication, and thus can include a group of students whoexperienced a similar type of error during a similar electroniccommunication.

In some embodiments, the step of block 1216 can further includeidentifying one or several user experience cohorts that are relevant toone or several of the current users. This identification can includegrouping one or several of the current users into a cohort based onerror correspondence and/or correspondence between the electroniccommunication and/or the subject matter of the electroniccommunications. And then identifying the correspondence between thegroup of one or several of the current users and one or several of theuser experience cohorts.

After the user experience cohorts have been identified, the process 1200proceeds to block 1218, wherein an expected user performance score isgenerated. In some embodiments, the expected user performance score canidentify the expected outcomes of the users in one or several of theuser experience cohorts. This expected user performance score can bebased on past user performance and/or on performance in parts of thecurrent test not affected by the error.

After the expected user performance score has been generated, theprocess 1200 proceeds to block 1220, wherein a deviation from theexpected score is identified. This deviation can be identified bycomparing a user's actual score to the user's expected user performancescore. After the deviation has been identified, the process 1200proceeds to block 1222, wherein the significance of this deviation canbe identified. The deviation can characterize the degree to which theactual tester score deviates from the user's expected user performancescore. In some embodiments, the significance of this deviation can beidentified according to one or several statistical methods.

After the significance of the deviation is identified, the process 1200proceeds to block 1224, wherein an experience adjustment value isidentified. In some embodiments, the experience adjustment value can bea characterization of the effect of the error on the user's performance,and can identify a degree to which the user's score should be modifiedto compensate for the error. In some embodiments, the adjustment valuecan be configured to improve the user's score based on the error.

After the adjustment value has been generated, the process 1200 proceedsto block 1226, wherein the score of the user, for whom the adjustmentvalue was generated, is modified with the adjustment value. In someembodiments, this modified score can be stored in the database 104, andcan be specifically stored in the user profile database 301. Further, insome embodiments, this modified score can be provided to one or severalusers.

With reference now to FIG. 13, a flowchart illustrating one embodimentof a process 1300 for receiving electronic communication systemperformance feedback is shown. The process 1300 begins at block 1302,wherein content such as a test is received. In some embodiments, thiscan include the receipt of testing content, and the storing of thetesting content in one of the databases 104 such as, for example, thecontent library database 303.

After the content has been received, the process 1300 proceeds to block1304, wherein log-in information for a plurality of users is received.In some embodiments, this information can be received as part of theinitiation of testing, and the log-in information can be received by,for example, the content management server 102 from one or several ofthe user devices 106. After the log-in information is received, theprocess 1300 proceeds to block 1306, wherein the log-in information isverified.

After the log-in information is verified, the process 1300 proceeds toblock 1308, wherein a first communication connection is established. Insome embodiments, the first communication connection can be between afirst server such as, for example, the content management server 102,and one or several of the user devices 106. After the firstcommunication connection has been established, the process 1300 proceedsto block 1310, wherein a second communication connection is established.In some embodiments, the second communication connection can beestablished between a second server such as, for example, the errorserver 119, and the one or several of the user devices 106 connected tothe first server via the first communication connection.

After the second communication connection has been established, theprocess 1300 proceeds to block 1312, wherein the content such as thetest and/or testing content is provided to the one or several userdevices 106 via the first communication connection. This can include thetransmission of electrical signals from the first server to the one orseveral user devices 106.

After the content has been provided, the process 1300 proceeds to block1314, wherein a feedback communication is received via the firstcommunication connection. In some embodiments, the feedbackcommunication can be received by the first server. In some embodiments,the feedback communication can include data identifying, for example,one or several errors in the communication of the content, theerror-free communication of the content, or the like. In someembodiments, the feedback communication received via the firstconnection can comprise on or several answers from, for example, theusers of the user devices 106 that received content in block 1312.

After the feedback communication has been received via the firstcommunication connection, the process 1300 proceeds to block 1316,wherein the feedback communication is received via the secondcommunication connection. In some embodiments, the feedbackcommunication can be received by the error server 119. In someembodiments, the feedback received by the second connection can comprisethe same data as contained in the feedback received via the firstconnection. In some embodiments, the feedback received by the secondconnection can comprise different data than contained in the feedbackreceived via the first connection. In some embodiments, for example, thefeedback received via the first connection can comprise answer data inresponse to test content, and the feedback received via the secondconnection does not include answer data. In some embodiments, thefeedback received via one or both of the first connection and the secondconnection can comprise error data.

After the feedback has been received via the second communicationconnection, the process 1300 proceeds to block 1318, wherein thefeedback is reconciled. In some embodiments, this can includeidentifying one or several error messages received via only one of thefirst and second communication connections. This reconciliation can, insome embodiments, indicate whether both the first and second connectionsreceived the same error messages. In some embodiments, this redundancycan facilitate in identifying one or several errors arising in the firstconnection. After the feedback has been reconciled, the process 1300proceeds to block 1320, and continues with block 1014 of FIG. 10.

A number of variations and modifications of the disclosed embodimentscan also be used. Specific details are given in the above description toprovide a thorough understanding of the embodiments. However, it isunderstood that the embodiments may be practiced without these specificdetails. For example, well-known circuits, processes, algorithms,structures, and techniques may be shown without unnecessary detail inorder to avoid obscuring the embodiments.

Implementation of the techniques, blocks, steps and means describedabove may be done in various ways. For example, these techniques,blocks, steps and means may be implemented in hardware, software, or acombination thereof. For a hardware implementation, the processing unitsmay be implemented within one or more application specific integratedcircuits (ASICs), digital signal processors (DSPs), digital signalprocessing devices (DSPDs), programmable logic devices (PLDs), fieldprogrammable gate arrays (FPGAs), processors, controllers,micro-controllers, microprocessors, other electronic units designed toperform the functions described above, and/or a combination thereof.

Also, it is noted that the embodiments may be described as a processwhich is depicted as a flowchart, a flow diagram, a swim diagram, a dataflow diagram, a structure diagram, or a block diagram. Although adepiction may describe the operations as a sequential process, many ofthe operations can be performed in parallel or concurrently. Inaddition, the order of the operations may be re-arranged. A process isterminated when its operations are completed, but could have additionalsteps not included in the figure. A process may correspond to a method,a function, a procedure, a subroutine, a subprogram, etc. When a processcorresponds to a function, its termination corresponds to a return ofthe function to the calling function or the main function.

Furthermore, embodiments may be implemented by hardware, software,scripting languages, firmware, middleware, microcode, hardwaredescription languages, and/or any combination thereof. When implementedin software, firmware, middleware, scripting language, and/or microcode,the program code or code segments to perform the necessary tasks may bestored in a machine-readable medium such as a storage medium. A codesegment or machine-executable instruction may represent a procedure, afunction, a subprogram, a program, a routine, a subroutine, a module, asoftware package, a script, a class, or any combination of instructions,data structures, and/or program statements. A code segment may becoupled to another code segment or a hardware circuit by passing and/orreceiving information, data, arguments, parameters, and/or memorycontents. Information, arguments, parameters, data, etc. may be passed,forwarded, or transmitted via any suitable means including memorysharing, message passing, token passing, network transmission, etc.

For a firmware and/or software implementation, the methodologies may beimplemented with modules (e.g., procedures, functions, and so on) thatperform the functions described herein. Any machine-readable mediumtangibly embodying instructions may be used in implementing themethodologies described herein. For example, software codes may bestored in a memory. Memory may be implemented within the processor orexternal to the processor. As used herein the term “memory” refers toany type of long term, short term, volatile, nonvolatile, or otherstorage medium and is not to be limited to any particular type of memoryor number of memories, or type of media upon which memory is stored.

Moreover, as disclosed herein, the term “storage medium” may representone or more memories for storing data, including read only memory (ROM),random access memory (RAM), magnetic RAM, core memory, magnetic diskstorage mediums, optical storage mediums, flash memory devices and/orother machine-readable mediums for storing information. The term“machine-readable medium” includes, but is not limited to portable orfixed storage devices, optical storage devices, and/or various otherstorage mediums capable of storing that contain or carry instruction(s)and/or data.

While the principles of the disclosure have been described above inconnection with specific apparatuses and methods, it is to be clearlyunderstood that this description is made only by way of example and notas limitation on the scope of the disclosure.

What is claimed is:
 1. A system, comprising: a memory including an errordatabase including information identifying a plurality of detectederrors and associated configuration information; and a server computer,wherein the server computer is configured to: receive an electricalsignal comprising attribute information identifying an attribute of auser device; receive an error message associated with an electroniccommunication between the server computer and the user device; identifya trend in error messages for the electronic communication between theserver computer and the user device by comparing the error message tothe plurality of detected errors and using the attribute of the userdevice; generate an adjustment value with which testing scores ofstudents can be adjusted using the trend in error messages; adjust atesting score of a student associated with the user device using theadjustment value to generate an adjusted testing score; and store theadjusted testing score in a score database.
 2. The system of claim 1,wherein the attribute information comprises location information,wherein the location information identifies the location of each of aplurality of user devices.
 3. The system of claim 1, wherein theelectronic communication comprises test communications.
 4. The system ofclaim 3, wherein the test communications comprise a plurality ofquestions and a plurality of responses to the plurality of questions. 5.The system of claim 1, wherein the attribute information compriseshardware information, wherein the hardware information identifieshardware of each of a plurality of user devices.
 6. The system of claim5, wherein the attribute information comprises software information,wherein the software information identifies software on each of theplurality of user devices.
 7. The system of claim 6, wherein theattribute information comprises an event log identifying operationsperformed by each of the plurality of user devices before the errormessage associated with the electronic communication was received by theserver computer and identifying software running on each of theplurality of user devices at a time the error message associated withthe electronic communication was received by the server computer.
 8. Thesystem of claim 1, wherein the server computer is further configured tosend an alert, wherein the alert is sent from the server computer to theuser device.
 9. The system of claim 8, wherein the alert launches anapplication within the user device, wherein the application displaysdata contained in the alert.
 10. A method, comprising: receiving, by aserver computer, an electrical signal comprising attribute informationidentifying an attribute of a user device; receiving, by the servercomputer, an error message associated with an electronic communicationbetween the server computer and the user device; identifying, by theserver computer, a trend in error messages for the electroniccommunication between the server computer and the user device bycomparing the error message to a plurality of detected errors stored inan error database; generating, by the server computer, an adjustmentvalue with which testing scores of students can be adjusted using thetrend in error messages; adjusting, by the server computer, a testingscore of a student associated with the user device using the adjustmentvalue to generate an adjusted testing score; and storing, by the servercomputer, the adjusted testing score in a score database.
 11. The methodof claim 10, wherein the attribute information comprises locationinformation, wherein the location information identifies the location ofeach of a plurality of user devices.
 12. The method of claim 10, whereinthe electronic communication comprises test communications including aplurality of questions and a plurality of responses to the plurality ofquestions.
 13. The method of claim 10, wherein the attribute informationcomprises hardware information, wherein the hardware informationidentifies hardware of each of a plurality of user devices.
 14. Themethod of claim 13, wherein the attribute information comprises softwareinformation, wherein the software information identifies software oneach of the plurality of user devices.
 15. The method of claim 14,wherein the attribute information comprises an event log identifyingoperations performed by each of the plurality of user devices before theerror message associated with the electronic communication was receivedby the server computer and identifying software running on each of theplurality of user devices at a time the error message associated withthe electronic communication was received by the server computer.
 16. Amethod, comprising: receiving, by a first server computer, an errormessage associated with an electronic communication between a secondserver computer and a user device; identifying, by the first servercomputer, a trend in error messages for the electronic communicationbetween the second server computer and the user device by comparing theerror message to a plurality of detected errors stored in an errordatabase; generating, by the first server computer, an adjustment valuewith which testing scores of students can be adjusted using the trend inerror messages; and transmitting, by the first server computer, theadjustment value to the second computer server.
 17. The method of claim16, wherein the electronic communication comprises test communicationsincluding a plurality of questions and a plurality of responses to theplurality of questions.
 18. The method of claim 16, further comprisingidentifying, by the first server computer an attribute of the userdevice by accessing an attribute database storing attribute informationfor a plurality of user devices, wherein the attribute information forthe plurality of user devices includes hardware information, wherein thehardware information identifies hardware of each of the plurality ofuser devices and identifying the trend in error messages using theattribute of the user device.
 19. The method of claim 18, wherein theattribute information comprises software information, wherein thesoftware information identifies software on each of the plurality ofuser devices.
 20. The method of claim 19, wherein the attributeinformation comprises an event log identifying operations performed byeach of the plurality of user devices before the error messageassociated with the electronic communication was received by the firstserver computer and identifying software running on each of theplurality of user devices at a time the error message associated withthe electronic communication was received by the first server computer.